Image processing apparatus and image processing method

ABSTRACT

The present disclosure relates to an image processing apparatus and an image processing method which make it possible to generate an image using texture images and depth images of two viewpoints that represent a three-dimensional structure of a greater number of regions. A drawing section generates a texture image of a predetermined viewpoint using a texture image obtained by perspectively projecting, to a perspective projection face perpendicular to a sight line direction heading from each of two viewpoints which are opposed to each other across a center of a polygon, toward the center of the polygon, a rear face of the polygon and a depth image corresponding to the texture image of each of the viewpoints. The present disclosure can be applied, for example, to a display apparatus and the like.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2017/027264 filed on Jul. 27, 2017, which claimspriority benefit of Japanese Patent Application No. JP 2016-157198 filedin the Japan Patent Office on Aug. 10, 2016. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus and animage processing method, and particularly to an image processingapparatus and an image processing method that make it possible torepresent a three-dimensional structure of a greater number of regionsusing texture images and depth images of two viewpoints.

BACKGROUND ART

As a technique for representing a three-dimensional structure of anobject, mainly a technique of representing a three-dimensional structureof an object using a polygon (3D mesh) of the object and a texturepasted to the polygon and a technique of representing athree-dimensional structure of an object using texture images and depthimages obtained by perspectively projecting the object with respect to aplurality of viewpoints are available. The former technique is atechnique used in a general CG (Computer Graphics) technology. Thelatter technique (hereinafter referred to as 2D depth technique) is highin affinity for a picked up image that is an image obtained byperspectively projecting an imaging object.

In a case where a three-dimensional structure of an object isrepresented by the 2D depth technique, data representing thethree-dimensional structure is encoded using an encoding method such asMPEG MVD (Moving Picture Experts Group phase Multi-view+depth) (forexample, refer to NPL 1).

CITATION LIST Non Patent Literature

-   [NPL 1]-   Takanori SENOH, Kenji YAMAMOTO, Ryutaro OI and Taiichiro KURITA    “MPEG Multi-View Image Coding Standardization,” Journal of the    National Institute of Information and Communications Technology Vol.    56 Nos. 1/2, issued March, 2010

SUMMARY Technical Problem

As described above, the 2D depth technique is a technique ofrepresenting a three-dimensional structure of an object using textureimages (two-dimensional images) obtained by perspectively projecting theobject with respect to a plurality of viewpoints and depth imagescorresponding to the texture images. Accordingly, the 2D depth techniquecannot represent a three-dimensional structure of a region to which theobject is not perspectively projected.

Therefore, by increasing the number of viewpoints for perspectiveprojection to increase regions to be perspectively projected, athree-dimensional structure of a greater number of regions can berepresented. However, as the number of viewpoints for perspectiveprojection increases, a data amount necessary to represent athree-dimensional structure increases.

The present disclosure has been made in view of such a situation asdescribed above and makes it possible to represent a three-dimensionalstructure of a greater number of regions using texture images and depthimages of two viewpoints.

Solution to Problem

An image processing apparatus of a first aspect of the presentdisclosure is an image processing apparatus including an imagegeneration section configured to generate a texture image of apredetermined viewpoint using a texture image obtained by projecting, toa projection face perpendicular to a sight line direction heading fromeach of two viewpoints which are opposed to each other across a centerof a polygon, toward the center of the polygon, a rear face of thepolygon and a depth image corresponding to the texture image of each ofthe viewpoints.

An image processing method of the first aspect of the present disclosurecorresponds to the image processing apparatus of the first aspect of thepresent disclosure.

In the first aspect of the present disclosure, a texture image of apredetermined viewpoint is generated using a texture image obtained byprojecting, to a projection face perpendicular to a sight line directionheading from each of two viewpoints which are opposed to each otheracross the center of a polygon, toward the center of the polygon, a rearface of the polygon and a depth image corresponding to the texture imageof each of the viewpoints.

The image processing apparatus of a second aspect of the presentdisclosure is an image processing apparatus including an imagegeneration section configured to generate a texture image by projecting,to a projection face perpendicular to a sight line direction headingfrom each of two viewpoints which are opposed to each other across acenter of a polygon, toward the center of the polygon, a rear face ofthe polygon and generate a depth image corresponding to the textureimage of each of the viewpoints.

An image processing method of the second aspect of the presentdisclosure corresponds to the image processing apparatus of the secondaspect of the present disclosure.

In the second aspect of the present disclosure, to a projection faceperpendicular to a sight line direction heading from each of twoviewpoints which are opposed to each other across the center of apolygon, toward the center of the polygon, a rear face of the polygon isprojected to generate a texture image, and a depth image thatcorresponds to the texture image of each of the viewpoints is generated.

It is to be noted that the image processing apparatus of the firstaspect and the second aspect of the present disclosure can beimplemented by causing a computer to execute a program.

Further, the program for being executed by a computer in order toimplement the information processing apparatus of the first aspect andthe second aspect of the present disclosure may be provided bytransmission through a transmission medium or by recording the programin a recording medium.

Advantageous Effects of Invention

According to the first aspect of the present disclosure, an image can begenerated. According to the first aspect of the present disclosure, animage can be generated using texture images and depth images of twoviewpoints representing a three-dimensional structure of a greaternumber of regions.

Further, according to the second aspect of the present disclosure, animage can be generated. According to the second aspect of the presentdisclosure, a three-dimensional structure of a greater number of regionscan be represented using texture images and depth images of twoviewpoints.

It is to be noted that the advantageous effects described here are notnecessarily restrictive and may be some advantageous effects describedin the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting a configuration example of a firstembodiment of a generation apparatus as an image processing apparatus towhich the present disclosure is applied.

FIG. 2 is a view depicting an arrangement example of an imagingapparatus.

FIGS. 3A, 3B, and 3C are views illustrating a texture image generated byperspectively projecting a front face of each polygon and a depth imagecorresponding to the texture image.

FIGS. 4A and 4B are views illustrating a texture image generated byperspectively projecting a front face of each polygon and a depth imagecorresponding to the texture image.

FIG. 5 is a view illustrating a texture image generated by perspectivelyprojecting a front face of each polygon and a depth image correspondingto the texture image.

FIG. 6 is a view illustrating a texture image generated by perspectivelyprojecting a rear face of a sphere and a depth image corresponding tothe texture image.

FIG. 7 is a view illustrating a texture image generated by perspectivelyprojecting a rear face of a sphere and a depth image corresponding tothe texture image.

FIGS. 8A, 8B, and 8C are views illustrating a texture image generated byperspectively projecting a rear face of each polygon and a depth imagecorresponding to the texture image.

FIGS. 9A and 9B are views illustrating a texture image generated byperspectively projecting a rear face of each polygon and a depth imagecorresponding to the texture image.

FIGS. 10A and 10B are views illustrating a texture image generated byperspectively projecting a rear face of each polygon and a depth imagecorresponding to the texture image.

FIG. 11 is a flow chart illustrating a generation process in thegeneration apparatus of FIG. 1.

FIG. 12 is a block diagram depicting a configuration example of a firstembodiment of a display apparatus as an image processing apparatus towhich the present disclosure is applied.

FIG. 13 is a view illustrating a first reconstruction method.

FIG. 14 is a view illustrating a second reconstruction method.

FIG. 15 is a flow chart illustrating a displaying process in the displayapparatus of FIG. 12.

FIG. 16 is a block diagram depicting a configuration example of a secondembodiment of a generation apparatus as an image processing apparatus towhich the present disclosure is applied.

FIG. 17 is a view illustrating a relationship between two viewpoints anda region capable of representing a three-dimensional structure.

FIGS. 18A and 18B are views illustrating a relationship between twoviewpoints and a region capable of representing a three-dimensionalstructure.

FIG. 19 is a view illustrating a first determination method ofcandidates for a pair of two viewpoints.

FIG. 20 is a view illustrating a second determination method ofcandidates for a pair of two viewpoints.

FIG. 21 is a view depicting an example of a table.

FIG. 22 is a flow chart illustrating a generation process of thegeneration apparatus of FIG. 16.

FIG. 23 is a flow chart illustrating the generation process of thegeneration apparatus of FIG. 16.

FIG. 24 is a view illustrating a different generation method of atexture image.

FIGS. 25A and 25B are views depicting a different example of a textureimage.

FIG. 26 is a view illustrating a coordinate system of a projection face.

FIG. 27 is a view illustrating tan axis projection.

FIG. 28 is a block diagram depicting a configuration example of hardwareof a computer.

FIG. 29 is a block diagram depicting an example of schematicconfiguration of a vehicle control system.

FIG. 30 is a diagram of assistance in explaining an example ofinstallation positions of an outside-vehicle information detectingsection and an imaging section.

DESCRIPTION OF EMBODIMENTS

In the following, a mode for carrying out the present disclosure(hereinafter referred to as embodiment) is described. It is to be notedthat the description is given in the following order.

1. First Embodiment: Generation Apparatus and Display Apparatus (FIGS.1, 2, 3A, 3B, 3C, 4A, 4B, 5, 6, 7, 8A, 8B, 8C, 9A, 9B, 10A, 10B, 11, 12,13, 14, and 15)

2. Second Embodiment: Generation Apparatus and Display Apparatus (FIGS.16, 17, 18A, 18B, 19, 20, 21, 22, and 23)

3. Different Generation Method of Texture Image (FIG. 24)

4. Different Example of Texture Image (FIGS. 25A and 25B)

5. Third Embodiment: tan Axis Projection (FIGS. 26 and 27)

6. Fourth Embodiment: Computer (FIG. 28)

7. Application Example (FIGS. 29 and 30)

1. First Embodiment

(Configuration Example of Generation Apparatus)

FIG. 1 is a block diagram depicting a configuration example of a firstembodiment of a generation apparatus as an image processing apparatus towhich the present disclosure is applied.

The generation apparatus 12 of FIG. 1 uses picked up images and depthimages acquired by imaging apparatuses 11-1 to 11-N (N is equal to orgreater than 2) to generate a texture image and a depth image of a mainimaging object in the picked up images and a texture image and a depthimage of an omnidirectional image.

In particular, the imaging apparatuses 11-1 to 11-N are arranged arounda main imaging object and include at least part of the main imagingobject in an imaging range thereof. Each of the imaging apparatuses 11-1to 11-N includes an picked up image camera 21 and a depth image camera22. The picked up image camera 21 images an imaging object to acquire apicked up image in a unit of a frame and supplies the picked up image tothe generation apparatus 12. The depth image camera 22 acquires aposition in a depth direction of the imaging object at each pixel of thepicked up image, generates a depth image in which informationrepresentative of the position is used as a pixel value, and suppliesthe depth image to the generation apparatus 12. It is to be noted that,in the following description, in a case where there is no necessity tospecifically distinguish the imaging apparatuses 11-1 to 11-N from eachother, they are collectively referred to as an imaging apparatus 11.

The generation apparatus 12 includes a region extraction section 31, aposition information generation section 32, a color informationgeneration section 33, a polygon generation section 34, a drawingsection 35, an omnidirectional image generation section 36, a resolutionreduction section 37, an encoder 38, a storage section 39 and atransmission section 40.

The region extraction section 31 of the generation apparatus 12 extractsa region of a main imaging object from N picked up images and N depthimages supplied from the N imaging apparatus 11 and supplies the regionsto the position information generation section 32. Further, the regionextraction section 31 extracts a region other than the region of themain imaging object as a background region from the N picked up imagesand the N depth images and supplies the background regions to theomnidirectional image generation section 36.

The position information generation section 32 uses the N depth imagesof the regions of the main imaging object supplied from the regionextraction section 31 to generate position information of one or morepolygons corresponding to the main imaging object. The positioninformation of a polygon is three-dimensional coordinates of vertices ofthe polygon in a 3D model coordinate system that is a three-dimensionalcoordinate system having an origin at a center of the main imagingobject. The position information generation section 32 supplies theposition information of each polygon to the color information generationsection 33 and the polygon generation section 34. Further, the positioninformation generation section 32 supplies the N picked up images in theregions of the main imaging object to the color information generationsection 33.

The color information generation section 33 uses position information ofeach polygon supplied from the position information generation section32 and the N picked up images of the regions of the main imaging objectto generate color information such as RGB values of a front face and arear face of each polygon. In particular, the color informationgeneration section 33 uses pixel values of the picked up imagescorresponding to the polygons to generate color information of the frontface of each of the polygons. Further, the color information generationsection 33 generates color information of the front face of each of thepolygons also as color information of the rear face of each of thepolygons. The color information generation section 33 supplies eachpiece of the color information of the front face and the rear face ofeach of the polygons to the polygon generation section 34.

It is to be noted that the color information of the front face of thepolygon is represented by describing three-dimensional coordinates in a3D model coordinate system of each vertex of the polygon in a clockwisedirection around a normal vector to the front face and describing colorinformation in an associated relationship with the three-dimensionalcoordinates. The color information of the rear face of the polygon isalso represented similarly to the color information of the front face.

The polygon generation section 34 generates each of the polygons on thebasis of the position information of each of the polygons supplied fromthe position information generation section 32 and pastes a texture toeach of the front face and the rear face of each of the polygons on thebasis of the color information of each of the front face and the rearface of each of the polygons supplied from the color informationgeneration section 33. The polygon generation section 34 supplies eachof the polygons having the texture pasted to the front face and the rearface of each of the polygons, to the drawing section 35.

The drawing section 35 (image generation section) performs perspectiveprojection of the rear face of each of the polygons to a perspectiveprojection face in regard to each of two viewpoints determined inadvance that are opposed to each other across the origin of the 3D modelcoordinate system that is the center of one or more polygons of the mainimaging object to generate texture images of the two viewpoints. Inparticular, the drawing section 35 perspectively projects, for each ofthe two viewpoints, the rear face of each polygon to a perspectiveprojection face that is a normal whose center is passed by a straightline in a sight line direction heading from each viewpoint to the originto generate texture images of the two viewpoints. In the presentspecification, an “opposed position” not only is an opposed position butalso includes the proximity to the opposed position within a rangewithin which a technical effect of the present disclosure is achieved.Similarly, the “normal” includes not only a normal itself but also aline having an angle to the face that is proximate to the vertical.

It is to be noted that, although a format of a texture image is notspecifically restricted, the YCbCr420 format can be adopted, forexample. The drawing section 35 generates depth images individuallycorresponding to texture images of two viewpoints on the basis ofpolygons. The drawing section 35 supplies the texture images of twoviewpoints and the depth images to the encoder 38.

The omnidirectional image generation section 36 perspectively projects Npicked up images of the background region supplied from the regionextraction section 31 to a regular octahedron centered at the origin ofthe 3D model coordinate system to generate a texture image of anomnidirectional image over 360 degrees around in a horizontal directionand 180 degrees around in a vertical direction. It is to be noted thatthe omnidirectional image may not be an image of an all space of asphere over 360 degrees around in a horizontal direction and 180 degreesaround in a vertical direction but may be an image of a partial space ifthe technological effect of the present disclosure is achieved. Theomnidirectional image generation section 36 perspectively projects Ndepth images of the background supplied from the region extractionsection 31 to the regular octahedron similarly to the picked up imagesto generate a depth image of an omnidirectional image. Theomnidirectional image generation section 36 supplies the texture imageand the depth image of the omnidirectional image to the resolutionreduction section 37.

The resolution reduction section 37 converts the texture image and thedepth image of the omnidirectional image supplied from theomnidirectional image generation section 36 into those of low resolutionand supplies them to the encoder 38.

The encoder 38 encodes the texture images of two viewpoints suppliedfrom the drawing section 35 and the depth images and encodes the textureimage and the depth image of the omnidirectional image supplied from theresolution reduction section 37. Although the AVC (Advanced VideoCoding) method, HEVC method, MVD method, or the like can be used as theencoding method for encoding of them, it is assumed that the AVC methodis used here.

Accordingly, the encoder 38 generates, by encoding, encoded streams ofthe texture images of the viewpoints (hereinafter referred to asviewpoint texture streams) and encoded streams of the depth images(hereinafter referred to as viewpoint depth streams). Further, theencoder 38 generates, by encoding, an encoded stream of the textureimage of the omnidirectional image of the reduced resolution(hereinafter referred to as an omnidirectional texture stream) and anencoded stream of the depth image of the omnidirectional image of thereduced resolution (hereinafter referred to as an omnidirectional depthstream). The encoder 38 supplies the viewpoint texture streams of twoviewpoints and the viewpoint depth streams as well as theomnidirectional texture stream and the omnidirectional depth stream tothe storage section 39.

The storage section 39 stores the viewpoint texture streams of twoviewpoints and the viewpoint depth streams as well as theomnidirectional texture stream and the omnidirectional depth streamsupplied thereto from the encoder 38.

The transmission section 40 reads out and transmits the viewpointtexture streams of two viewpoints stored in the storage section 39 andviewpoint depth streams as well as the omnidirectional texture streamand the omnidirectional depth stream.

The generation apparatus 12 converts polygons and color informationrepresentative of a three-dimensional structure of a main imaging objectinto texture images and depth images of two viewpoints in such a manneras described above. Accordingly, the generation apparatus 12 can encodethe texture images and the depth images of two viewpoints using agenerally used image encoding method to reduce the data amount of them.As a result, a transmission bandwidth of data representing thethree-dimensional structure of the main imaging object can be reduced.

It is to be noted that, although the generation apparatus 12 of FIG. 1generates the polygons and color information, the generation apparatus12 may generate some other information of a point cloud or the like aslong as it is information that represents a three-dimensional structureused in the CG technology.

Further, while, in the example of FIG. 1, the depth image camera 22acquires a depth image of a pixel number equal to that of a picked upimage, in a case where the depth image camera 22 acquires a depth imageof a pixel number smaller than that of a picked up image, a depth imageinterpolation section interpolating pixel values of a depth image isprovided between the region extraction section 31 and the positioninformation generation section 32. In this case, the depth imageinterpolation section interpolates pixel values of a depth image to makethe pixel number of the depth image equal to the pixel number of thepicked up image.

Furthermore, while, in the example of FIG. 1, each imaging apparatus 11acquires a depth image, the depth image may be generated from picked upimages acquired from an imaging apparatus 11 corresponding to the depthimage and a different imaging apparatus 11.

(Arrangement Example of Image Pickup Apparatus)

FIG. 2 is a view depicting an arrangement example of the imagingapparatus 11 of FIG. 1.

In the example of FIG. 2, N is 9.

As depicted in FIG. 2, the nine imaging apparatuses 11-1 to 11-9 arearranged so as to surround a main imaging object 61.

(Description of Effect)

FIGS. 3A, 3B, 3C, 4A, 4B, and 5 are views illustrating, for each of twoviewpoints opposing to each other across the origin of a 3D modelcoordinate system, a texture image generated by perspectively projectingthe front face of each polygon to a perspective projection face and adepth image corresponding to the texture image.

In the examples of FIGS. 3A, 3B, 3C, 4A, 4B, and 5, a polygon of a mainimaging object forms a sphere 81. In this instance, as depicted in FIG.5A, in a texture image generated by perspectively projecting a frontface of the sphere 81 with respect to a perspective projection face in asight line direction V1 from the viewpoint O1 that is one of the twoviewpoints, a texture pasted to a region 81A on the front face of thesphere 81 that crosses first with each of projection directions 82 isdrawn. The projection directions are directions that extend from theviewpoint and are within a range in which an absolute value of an angledefined with respect to the sight line direction is equal to one-half ofa field angle (θ, in the example of FIGS. 3A, 3B, and 3C). Further, thedepth image corresponding to the texture image is an imagerepresentative of the position in the depth direction (sight linedirection V1) of the region 81A with respect to the viewpoint O1.

Further, as depicted in FIG. 3B, in a texture image generated byperspectively projecting the front face of the sphere 81 to aperspective projection face in a sight line direction V2 from theviewpoint O2 that is the other one of the two viewpoints, a texturepasted to a region 81B on the front face of the sphere 81 that crossesfirst with each of projection directions 83 is drawn. Further, the depthimage corresponding to the texture image is an image representative ofthe position in the depth direction (sight line direction V2) of theregion 81B with respect to the viewpoint O2.

Accordingly, as depicted in FIG. 3C, the texture image and the depthimage from the viewpoint O1 and the texture image and the depth imagefrom the viewpoint O2 can be used to represent a three-dimensionalstructure of the two regions 81A and 81B opposing to each other across acenter of the sphere 81. However, on the surface of the sphere, a regionother than the region 81A and the region 81B exists. In other words, aregion of the surface of the sphere 81 with which a three-dimensionalstructure cannot be represented by the texture image and the depth imagefrom the viewpoint O1 and the texture image and the depth image from theviewpoint O2 exists.

For example, in a case where a world map is applied as a texture to thefront face and a rear face of the sphere 81 and the sky above theAtlantic Ocean off the coast of Africa is the viewpoint O1, part of theAfrican Continent and the South American Continent pasted as a textureto the front face of the region 81A is drawn in a texture image 101 ofthe viewpoint O1 as depicted on the left side in FIG. 4A.

Further, in this case, the viewpoint O2 is in the sky above the PacificOcean, and part of the Australian Continent pasted as a texture to thefront face of the region 81B is drawn in a texture image 102 of theviewpoint O2 as depicted on the left side in FIG. 4B. However, theAntarctica and the like are not drawn in any of the texture image 101and the texture image 102.

Further, as depicted on the right side in FIG. 4A and on the right sidein FIG. 4B, a depth image 111 corresponding to the texture image 101 andanother depth image 112 corresponding to the texture image 102 are same.It is to be noted that, in a depth image, as the distance of theposition in the depth direction of an imaging object at each pixelincreases, the pixel value (luminance value) decreases. Accordingly, inthe depth image 111 and the depth image 112, the pixel value is highestat the center and decreases as the distance from the center increases.

In this manner, in any of the texture image 101 and the texture image102, the Antarctica and the like are not drawn. Accordingly, as depictedin FIG. 5, a three-dimensional structure 121 reconstructed using thetexture image 101 and the depth image 111 as well as the texture image102 and the depth image 112 is only part of the sphere 81 having theworld map pasted to the front face and the rear face as a texture.

While, in the example of FIGS. 3A, 3B, 3C, 4A, 4B, and 5, the shape ofthe polygon is the comparatively simple sphere 81, in a case where theshape of the polygon is complicated, a region of the polygon whosethree-dimensional structure cannot be represented by texture images oftwo viewpoints increases.

FIGS. 6 and 7 are views illustrating a texture image generated byperspectively projecting the rear face of the sphere 81 with respect tothe viewpoint O1 to a perspective projection face in the sight linedirection V1 and a depth image corresponding to the texture image.

As described hereinabove, in a case where a texture image is generatedby perspectively projecting the front face of the sphere 81 to theperspective projection face in the sight line direction V1 with respectto the viewpoint O1, a texture applied to each of points c1 on the frontface of the sphere 81 with which each of the projection directions 82first crosses is drawn in the texture image as depicted in FIGS. 8A, 8B,and 8C. Further, the depth image corresponding to this texture image isan image representative of the position in the depth direction (sightline direction V1) at each point c1 with respect to the viewpoint O1.

On the other hand, in a case where a texture image is generated byperspectively projecting the rear face of the sphere 81 to theperspective projection face in the sight line direction V1 with respectto the viewpoint O1, a texture applied to each of the points c2 on therear face of the sphere 81 with which each of the projection directions82 first crosses is drawn in the texture image as depicted in FIGS. 8A,8B, and 8C. Further, the depth image corresponding to this texture imageis an image representative of the position in the depth direction (sightline direction V1) at each point c2 with respect to the viewpoint O2.

For example, in a case where a world map is pasted as a texture to thefront face and the rear face of the sphere 81 and the sky above theAtlantic Ocean off the coast of Africa is the viewpoint O1, as depictedin FIGS. 9A and 9B, to a texture image 131 of the viewpoint O1, theNorth American Continent, part of the South American Continent, theAntarctica, part of the European Continent, the Asian Continent and theAustralian Continent pasted as a texture to the rear face of each of thepoints c2 are drawn. Further, in a depth image 132 corresponding to thetexture image 131, the pixel value is lowest at the center and decreasesas the distance from the center increases.

FIGS. 8A, 8B, 8C, 9A, 9B, 10A, 10B, and 10C are views illustrating, foreach of two viewpoints opposing to each other across the origin of a 3Dmodel coordinate system, a texture image generated by perspectivelyprojecting the rear face of a polygon to a perspective projection faceand a depth image corresponding to the texture image.

In the example of FIGS. 8A, 8B, 8C, 9A, 9B, 10A, 10B, and 10C, thepolygon of the main imaging object is a sphere 81. In this case, asdepicted in A of FIG. 8A, to a texture image generated by perspectivelyprojecting the rear face of the sphere 81 to a perspective projectionface in a sight line direction V1 with respect to the viewpoint O1, atexture pasted to a region 153A of the rear face on the sphere 81 withwhich each of the projection directions 82 crosses first is drawn.Further, a depth image corresponding to the texture image is an imagerepresentative of the position in the depth direction (sight linedirection V1) of the region 81A with respect to the viewpoint O1.

Meanwhile, as depicted in FIG. 8B, to a texture image generated byperspectively projecting the rear face of the sphere 81 to a perspectiveprojection face in a sight line direction V2 with respect to the otherviewpoint O2 of the two viewpoints, a texture pasted to a region 153B ofthe rear face on the sphere 81 with which each of the projectiondirections 83 crosses first is drawn. Further, a depth imagecorresponding to the texture image is an image representative of theposition in the depth direction (sight line direction V2) of the region81B with respect to the viewpoint O2.

Accordingly, as depicted in FIG. 8C, a three-dimensional structure ofthe two regions 153A and 153B opposed to each other across the center ofthe sphere 81 can be represented with the texture image and the depthimage of the viewpoint O1 and the texture image and the depth image ofthe viewpoint O2.

It is to be noted that, as depicted in FIG. 8C, the region 153A and theregion 153B overlap with each other. Accordingly, a three-dimensionalstructure of the entire sphere 81 can be represented with the textureimage and the depth image of the viewpoint O1 and the texture image andthe depth image of the viewpoint O2.

For example, in a case where a world map is applied as a texture to thefront face and the face of the sphere 81 and the sky above the AtlanticOcean off the coast of Africa is the viewpoint O1, the North AmericanContinent, part of the South American Continent, the Antarctica, part ofthe European Continent, the Asian Continent and the Australian Continentpasted as a texture to the rear face of the region 153A are drawn in atexture image 161 of the viewpoint O1 as depicted on the left side inFIG. 9A.

Further, in this case, the viewpoint O2 is in the sky above the PacificOcean, and the African Continent, the North American Continent, theSouth American Continent, the Antarctica and part of the EuropeanContinent pasted as a texture to the rear face of the region 153B aredrawn in a texture image 162 of the viewpoint O2 as depicted on the leftside in FIG. 9B. Accordingly, all of the seven continents are drawn inat least one of the texture image 161 and the texture image 162.

Further, as depicted on the right side in FIG. 9A and FIG. 9B, a depthimage 163 corresponding to the texture image 161 and a depth image 164corresponding to the texture image 162 are same. In the depth image 163and the depth image 164, the pixel value is highest at the center anddecreases as the distance from the center increases.

All of the seven continents are drawn in at least one of the textureimage 161 and the texture image 162 in such a manner as described above.Accordingly, as depicted in FIG. 10A, a three-dimensional structure 171reconstructed using the texture image 161 and the depth image 163 is aportion greater than one-half of the sphere 81 on the viewpoint O2 side(right half in the figure). Meanwhile, as depicted in FIG. 10B, athree-dimensional structure 172 reconstructed using the texture image162 and the depth image 164 is a portion greater than one half thesphere 81 on the viewpoint O1 side (left half in the figure). Therefore,by reconstructing the three-dimensional structures using the textureimage 161 and the depth image 163 as well as the texture image 162 andthe depth image 164, the entire sphere 81 can be generated.

It is to be noted that the overlapping portion of the region 153A andthe region 153B is generated using the texture image 161 and the depthimage 163 or the texture image 162 and the depth image 164.

For example, as depicted in FIG. 8A, in a case where each region 154 ofend portions of the region 153A within the overlapping region of theregion 153A and the region 1538 is perspectively projected with respectto the viewpoint O1, the angle with respect to a projection direction 82is small. Accordingly, with the texture image 161 and the depth image163, a three-dimensional structure of the region 154 can be representedwith high accuracy.

However, in a case where the region 154 is perspectively projected withrespect to the viewpoint O2 as depicted in FIG. 8B, the angle of theregion 154 with respect to a projection direction 83 is greater incomparison with that in a case where it is perspectively projected withrespect to the viewpoint O1. Accordingly, with the texture image 162 andthe depth image 164, a three-dimensional structure of the region 154 canbe represented with higher accuracy in comparison with the texture image161 and the depth image 163. Therefore, the region 154 is generatedusing the texture image 162 and the depth image 164.

By generating the overlapping region of the region 153A and the region153B using one of the texture image 161 and the depth image 163 and thetexture image 162 and the depth image 164, which corresponds to aprojection direction having a greater angle with respect to theoverlapping region, the accuracy in reconstruction of the sphere 81 canbe improved.

(Description of Processing of Generation Apparatus)

FIG. 11 is a flow chart illustrating a generation process by thegeneration apparatus 12 of FIG. 1. This generation process is performedfor each of frames of N picked up images and N depth images acquired bythe N imaging apparatus 11.

At step S11 of FIG. 11, the region extraction section 31 of thegeneration apparatus 12 extracts a region of a main imaging object and abackground region from N picked up images and N depth images suppliedfrom the imaging apparatus 11. The region extraction section 31 suppliesthe N picked up images and depth images in the region of the mainimaging object to the position information generation section 32 andsupplies the N picked up images and depth images in the backgroundregion to the omnidirectional image generation section 36.

At step S12, the position information generation section 32 uses the Ndepth images in the region of the main imaging object supplied from theregion extraction section 31 to generate position information of each ofpolygons of the main imaging object and supplies the positioninformation to the color information generation section 33 and thedrawing section 35. Further, the position information generation section32 supplies the N picked up images in the region of the main imagingobject to the color information generation section 33.

At step S13, the color information generation section 33 uses theposition information of each of the polygons and the N picked up imagesin the region of the main imaging object supplied from the positioninformation generation section 32 to generate color information of thefront face and the rear face of each of the polygons. The colorinformation generation section 33 supplies the color information of thefront face and the rear face of each of the polygons to the drawingsection 35.

At step S14, the drawing section 35 generates polygons on the basis ofthe position information of each of the polygons supplied from theposition information generation section 32 and pastes a texture to thefront face and the rear face of each of the polygons on the basis of thecolor information of the front face and the rear face of each of thepolygons supplied from the color information generation section 33.

At step S15, the drawing section 35 generates texture images of twoviewpoints determined in advance by perspectively projecting, for eachof the two viewpoints, the rear face of each of the polygons to aperspective projection face in the sight line direction. The drawingsection 35 supplies the texture images of two viewpoints to the encoder38.

At step S16, the drawing section 35 generates depth images individuallycorresponding to the texture images of two viewpoints on the basis ofthe polygons and supplies the depth images to the encoder 38.

At step S17, the omnidirectional image generation section 36 generates atexture image of an omnidirectional image by perspectively projectingthe N picked up images in the background region supplied from the regionextraction section 31 to a regular octahedron centered at the origin ofthe 3D model coordinate system and supplies the texture image of theomnidirectional image to the resolution reduction section 37.

At step S18, the omnidirectional image generation section 36perspectively projects the N depth images in the background regionsupplied from the region extraction section 31 to the regular octahedronsimilarly to the picked up images to generate a depth image of anomnidirectional image and supplies the depth image of theomnidirectional image to the resolution reduction section 37.

At step S19, the resolution reduction section 37 converts the textureimage and the depth image of the omnidirectional image supplied from theomnidirectional image generation section 36 into those of a lowerresolution and supplies the texture image and the depth image of theomnidirectional image of the reduced resolution to the encoder 38.

At step S20, the encoder 38 encodes the texture images and the depthimages of two viewpoints supplied from the drawing section 35 and thetexture image and the depth image of an omnidirectional image suppliedfrom the resolution reduction section 37. The encoder 38 suppliesviewpoint texture streams and viewpoint depth streams of two viewpointsas well as an omnidirectional texture stream and an omnidirectionaldepth stream generated as a result of the encoding to the storagesection 39 so as to be stored.

At step S21, the transmission section 40 reads out the viewpoint texturestreams of two viewpoints and the viewpoint depth streams as well as theomnidirectional texture stream and the omnidirectional depth streamstored in the storage section 39 and transmits them. Then, theprocessing ends.

The generation apparatus 12 generates texture images and depth images oftwo viewpoints by perspectively projecting, for each of the twoviewpoints opposed to each other across the origin of the 3D modelcoordinate system, the rear face of a polygon to the perspectiveprojection face in the sight line direction of each viewpoint in such amanner as described above. Accordingly, the generated texture images anddepth images of two viewpoints can represent a three-dimensionalstructure of a polygon of a main imaging object in a greater number ofregions in comparison with an alternative case in which they aregenerated by perspectively projecting the front face of the polygon.

(Configuration Example of Display Apparatus)

FIG. 12 is a block diagram depicting a configuration example of thefirst embodiment of a display apparatus as an image processing apparatusto which the present disclosure is applied.

The display apparatus 200 of FIG. 12 receives viewpoint texture streamsof two viewpoints and viewpoint depth streams as well as anomnidirectional texture stream and an omnidirectional depth streamtransmitted from the generation apparatus 12 of FIG. 1 to generate atexture image of a predetermined viewpoint.

In particular, the display apparatus 200 includes a reception section201, a storage section 202, a decoder 203, a reconstruction section 204,a drawing section 205, and a display section 206.

The reception section 201 of the display apparatus 200 receivesviewpoint texture streams and viewpoint depth streams of two viewpointsas well as the omnidirectional texture stream and the omnidirectionaldepth stream transmitted from the generation apparatus 12 and suppliesthem to the storage section 202.

The storage section 202 stores the viewpoint texture streams and theviewpoint depth streams of two viewpoints as well as the omnidirectionaltexture stream and the omnidirectional depth stream supplied from thereception section 201.

The decoder 203 reads outs the viewpoint texture streams and viewpointdepth streams of two viewpoints as well as the omnidirectional texturestream and the omnidirectional depth stream from the storage section 202and decodes them. The decoder 203 supplies the texture images and thedepth images of two viewpoints as well as the omnidirectional textureimage and omnidirectional depth image obtained as a result of thedecoding to the reconstruction section 204.

The reconstruction section 204 reconstructs a three-dimensionalstructure of a main imaging object in the 3D model coordinate systemusing the pixels of the texture images and the depth images of twoviewpoints supplied to the decoder 203. As described hereinabove, thetexture images and the depth images of two viewpoints generated by thegeneration apparatus 12 can represent a three-dimensional structure of apolygon of a main imaging object in a greater number of regions incomparison with an alternative case in which they are generated byperspectively projecting the front face of the polygon. Accordingly, thenumber of regions of a main imaging object in which a three-dimensionalstructure is reconstructed using the decoded texture images and thedepth images of two viewpoints is greater than that in an alternativecase in which the texture images and the depth images of two viewpointsare generated by perspectively projecting a front face of a polygon.

Further, the reconstruction section 204 reconstructs a three-dimensionalstructure of the background region in the 3D model coordinate systemusing the texture image and the depth image of the omnidirectional imagesupplied from the decoder 203. The reconstruction section 204 suppliesthe position information and the color information of thethree-dimensional structures of the main imaging objects and thebackground region to the drawing section 205.

The drawing section 205 (image generation section) generates, on thebasis of the position information and the color information of thethree-dimensional structures of the main imaging object and thebackground region supplied from the reconstruction section 204, atexture image of the viewpoint, the sight line direction, and the fieldangle in the 3D model coordinate system designated by a viewer or thelike as a display image. The drawing section 205 supplies the generateddisplay image to the display section 206.

The display section 206 displays the display image supplied from thedrawing section 205. Consequently, the viewer can view the main imagingobject from an arbitrary position, for example, around the main imagingobject.

(Description of First Reconstruction Method)

FIG. 13 is a view illustrating a first reconstruction method.

Note that it is assumed that, in the example of FIG. 13, the resolutionof texture images and depth images of two viewpoints is 4 (horizontal)×4(vertical) pixels for the convenience of description. Further, FIG. 13illustrates a case in which a three-dimensional structure of a mainimaging object is reconstructed using a texture image and a depth imageof one viewpoint O1 of the two viewpoints.

The first reconstruction method is a method of reconstructing athree-dimensional structure using a point cloud. In particular, asdepicted in FIG. 13, according to the first reconstruction method, thereconstruction section 204 generates, on the basis of a viewpoint O1, asight line direction V1, a field angle 2θ, a position (u, v) of asampling point 231, which corresponds to each pixel 221 of a textureimage 220 of the viewpoint O1, on the texture image 220, and a pixelvalue of each pixel 221 of a depth image corresponding to the textureimage 220, three-dimensional coordinates (X, Y, Z) of the sampling point231 in a 3D model coordinate system.

Further, the reconstruction section 204 converts YCbCr values that are apixel value of each pixel 221 of the texture image 220 into RGB valuesand determines them as RGB values of the sampling point 231corresponding to the pixel 221. The reconstruction section 204 drawspoints of the RGB values of the sampling points 231 to thethree-dimensional coordinate values (X, Y, Z) of the sampling points 231to reconstruct a three-dimensional structure of the main imaging object.The reconstruction section 204 supplies the three-dimensionalcoordinates (X, Y, Z) of the sampling points 231 as position informationof the three-dimensional structures of the main imaging objects to thedrawing section 205 and supplies the RGB values of the sampling points231 as color information of the three-dimensional structures of the mainimaging objects to the drawing section 205.

(Description of Second Reconstruction Method)

FIG. 14 is a view illustrating a second reconstruction method.

Note that it is assumed that, in the example of FIG. 14, the resolutionof texture images and depth images of two viewpoints is 4 (horizontal)×3(vertical) pixels for the convenience of description. Further, FIG. 14depicts a case in which a three-dimensional structure of a main imagingobject is reconstructed using a texture image and a depth image of oneviewpoint O1 of two viewpoints.

The second reconstruction method is a method of reconstructing athree-dimensional structure using a triangle patch. In particular, asdepicted on the left side in FIG. 14, in the second reconstructionmethod, the reconstruction section 204 generates sampling points 251corresponding to pixels 241 on a texture image 240 of the viewpoint O1.The reconstruction section 204 connects three neighboring samplingpoints 251 from among the sampling points 251 corresponding to allpixels of the texture image 240 to generate triangle patches 252 havingvertices at the three neighboring sampling points 251.

Then, the reconstruction section 204 generates, on the basis of theviewpoint O1, a sight line direction V1, a field angle 2θ, a position(u, v) of each sampling point 251 on the texture image 240, and a pixelvalue of each pixel 241 of a depth image corresponding to the textureimage 240, three-dimensional coordinates (X, Y, Z) of the sampling point251 in a 3D model coordinate system.

Then, the reconstruction section 204 arranges, as depicted on the rightside in FIG. 14, sampling points 261 corresponding to the samplingpoints 251 in the 3D model coordinate system on the basis of thethree-dimensional coordinates (X, Y, Z) of the sampling points 251.Further, the reconstruction section 204 connects sampling points 261corresponding to three sampling points 251 configuring the vertices ofthe triangle patches 252 to generate triangle patches 262.

Further, the reconstruction section 204 converts, for each trianglepatch 262, YCbCr values of the pixels 241 configuring the triangle patch252 corresponding to the triangle patch 262 into RGB values and uses theRGB values to generate RGB values of the triangle patch 262. Thereconstruction section 204 pastes, for each triangle patch 262, atexture of RGB values of the triangle patch 262 to the triangle patch262. The reconstruction section 204 thereby reconstructs athree-dimensional structure of a main imaging object. The reconstructionsection 204 supplies the three-dimensional coordinates (X, Y, Z) of thesampling points 261 that are vertices of the triangle patches 262 aspositional information of the three-dimensional structures of a mainimaging object to the drawing section 205. Further, the reconstructionsection 204 supplies the RGB values of the triangle patches 262 as colorinformation of the three-dimensional structures of the main imagingobject to the drawing section 205.

While methods for reconstructing a three-dimensional structure of a mainimaging object from a texture image and a depth image of the viewpointO1 are described with reference to FIGS. 13 and 14, also a method forreconstructing a three-dimensional structure of a main imaging objectfrom a texture image and a depth image of the viewpoint O2 and a methodfor reconstructing a three-dimensional structure of a background regionfrom a texture image and a depth image of an omnidirectional image aresimilar to those just described.

(Description of Processing of Display Apparatus) FIG. 15 is a flow chartillustrating a display process of the display apparatus 200 of FIG. 12.This display process is started, for example, when a request to displaya display image is issued by a viewer in a state in which the viewpointtexture streams and the viewpoint depth streams of two viewpoints aswell as the omnidirectional texture stream and the omnidirectional depthstream are stored in the storage section 202.

At step S32 of FIG. 15, the decoder 203 reads out and decodes theviewpoint texture streams and the viewpoint depth streams of twoviewpoints as well as the omnidirectional texture stream and theomnidirectional depth stream from the storage section 202. The decoder203 supplies texture images and depth images of two viewpoints and anomnidirectional texture image and an omnidirectional depth imageobtained as a result of the decoding to the reconstruction section 204.

At step S33, the reconstruction section 204 reconstructs athree-dimensional structure of a main imaging object in a 3D modelcoordinate system using the texture images and the depth images of twoviewpoints supplied from the decoder 203. The reconstruction section 204supplies position information and color information of thethree-dimensional structure of a main imaging object to the drawingsection 205.

At step S34, the reconstruction section 204 reconstructs athree-dimensional structure of the background region in the 3D modelcoordinate system using the texture image and the depth image of anomnidirectional image supplied from the decoder 203. The reconstructionsection 204 supplies the position information and the color informationof the three-dimensional structure of the background region to thedrawing section 205.

At step S35, the drawing section 205 generates, on the basis of theposition information and the color information of the three-dimensionalstructures of the main imaging object and the background region suppliedfrom the reconstruction section 204, a texture image having theviewpoint, a sight line direction, and a field angle in the 3D modelcoordinate system designated by the viewer or the like as a displayimage. The drawing section 205 supplies the generated display image tothe display section 206.

At step S36, the display section 206 displays the display image suppliedfrom the drawing section 205 and ends the processing.

The display apparatus 200 generates a display image using texture imagesand depth images of two viewpoints generated by the generation apparatus12 in such a manner as described above. Accordingly, in comparison withan alternative case in which texture images and depth images of twoviewpoints generated by perspectively projecting the front face of apolygon for each of the two viewpoints, it is possible to reconstruct athree-dimensional structure of a main viewing object in a greater numberof regions and generate a display image from the three-dimensionalstructure. As a result, the picture quality of the display image isenhanced.

Second Embodiment

(Configuration Example of Generation Apparatus)

FIG. 16 is a block diagram depicting a configuration example of a secondembodiment of the generation apparatus as an information processingapparatus to which the present disclosure is applied.

From among components depicted in FIG. 16, like components to those ofFIG. 1 are denoted by like reference characters. Overlapping descriptionis suitably omitted.

The configuration of the generation apparatus 300 of FIG. 16 isdifferent from the configuration of the generation apparatus 12 of FIG.1 in that a viewpoint controlling section 301 is provided newly and thata drawing section 302, a storage section 303, and a transmission section304 are provided in place of the drawing section 35, the storage section39, and the transmission section 40. In the generation apparatus 300,the positions of two viewpoints are variable, and the positions of twoviewpoints are set such that texture images and depth images of the twoviewpoints represent a three-dimensional structure of a greatest numberof regions of polygons of a main imaging object.

In particular, the viewpoint controlling section 301 (viewpointinformation generation section) rotates a straight line, whichinterconnects a current pair of two viewpoints opposed to each otheracross the origin of the 3D model coordinate system and passes theorigin, successively by a predetermined amount in a predetermineddirection around the origin within a predetermined range to determine aplurality of candidates for a pair of two viewpoints including thecurrent pair of two viewpoints. A generation frequency of candidates fora pair of two viewpoints is not particularly restricted and can bedetermined for each frame, for each sequence, for each GOP (Group ofPicture) or the like. The viewpoint controlling section 301 generatesviewpoint information representative of three-dimensional coordinates ofa plurality of candidates for a pair of two viewpoints and supplies theviewpoint information to the drawing section 302.

Further, in a case where the candidate for a pair of two viewpointscorresponding to viewpoint information supplied from the drawing section302 is not the current pair of two viewpoints, the viewpoint controllingsection 301 changes the current pair of two viewpoints to the candidatefor a pair of two viewpoints and generates a table including theviewpoint information of the pair of two viewpoints. The viewpointcontrolling section 301 outputs the table to the storage section 303.

In the case where a plurality of pieces of viewpoint information aresupplied from the viewpoint controlling section 301, the drawing section302 (image generation section) perspectively projects, for each piece ofviewpoint information, the rear face of each polygon generated by thepolygon generation section 34 for each viewpoint of the candidates for apair of two viewpoints whose three-dimensional coordinates are indicatedby the viewpoint information to a perspective projection face togenerate texture images of the candidates for a pair of two viewpoints.

Then, the drawing section 302 selects, from among the candidates for apair of two viewpoints, a candidate for a pair of two viewpoints inwhich the region of the rear face of a polygon perspectively projectedupon generation of a texture image is greatest as an optimum pair of twoviewpoints. In particular, the drawing section 302 selects, from amongthe candidates for a pair of two viewpoints, a pair of two viewpoints inwhich the number of polygons whose rear face is perspectively projectedis greatest when texture images of two viewpoints are generated as anoptimum pair of two viewpoints.

The drawing section 302 retains viewpoint information of the optimumpair of two viewpoints as viewpoint information of the current pair oftwo viewpoints and supplies the viewpoint information to the viewpointcontrolling section 301. Further, the viewpoint controlling section 301determines the texture images of the optimum pair of two viewpoints asfinal texture images of a current pair of two viewpoints.

Conversely, in a case where a plurality of pieces of viewpointinformation are not supplied from the viewpoint controlling section 301,the drawing section 302 perspectively projects the rear face of eachpolygon for each viewpoint of the current pair of two viewpoints whosethree-dimensional coordinates are indicated by the retained viewpointinformation to a perspective projection face to generate texture imagesof the current pair of two viewpoints.

The drawing section 302 generates, on the basis of each polygon, depthimages individually corresponding to the texture images of the currentpair of two viewpoints. The drawing section 302 supplies the textureimages and the depth images of the current pair of two viewpoints to theencoder 38.

The storage section 303 stores the viewpoint texture streams and theviewpoint depth streams of two viewpoints and the omnidirectionaltexture stream and the omnidirectional depth stream supplied from theencoder 38. Further, the storage section 303 stores the table suppliedfrom the viewpoint controlling section 301.

The transmission section 304 reads out and transmits the viewpointtexture streams and the viewpoint depth streams of two viewpoints, theomnidirectional texture stream and the omnidirectional depth stream, andthe table stored in the storage section 39.

(Description of Relationship Between Two Viewpoints and Region Capableof Representing Three-Dimensional Structure)

FIGS. 17, 18A, and 18B are views illustrating a relationship between twoviewpoints and a region capable of representing a three-dimensionalstructure.

In a case where a polygon of a main imaging object forms a robot 320 ofFIG. 17, a schematic view taken along an aa′ cross section that ishorizontal with respect to an installation face for the robot 320 issuch as depicted in FIGS. 18A and 18B. In particular, the aa′ crosssection of the robot 320 includes a left arm 321, a torso 322, and aright arm 323 of the robot 320.

In this case, when one viewpoint O1 from between the two viewpoints isdisposed on the left side of the left arm 321 on the aa′ cross sectionand the other viewpoint O2 is disposed on the right side of the rightarm 323 as depicted in FIG. 18A, then the rear face of the right side ofthe left arm 321 is perspectively projected to the texture image of theviewpoint O1. Meanwhile, the rear face of the left side of the right arm323 is perspectively projected to the texture image of the viewpoint O2.

However, not only in the texture image of the viewpoint O1 but also inthe texture image of the viewpoint O2, the rear face of the torso 322 isblocked by the left arm 321 or the right arm 323 existing in front andis not perspectively projected.

In contrast, when the viewpoint O1′ from between the two viewpoints isdisposed on the upper side of the left arm 321, the torso 322, and theright arm 323 on the aa′ cross section and the other viewpoint O2′ isdisposed on the lower side as depicted in FIG. 18B, then to the textureimage of the viewpoint O1′, the rear face on the lower side of each ofthe left arm 321, torso 322 and right arm 323 is perspectivelyprojected. Meanwhile, to the texture image of the viewpoint O2′, therear face on the upper side of each of the left arm 321, the torso 322,and the right arm 323 is perspectively projected.

As described above, in a case where a polygon of a main imaging objectis configured from polygons of a plurality of portions, when polygons ofdifferent portions are lined up in the depth direction of each of thetwo viewpoints, then the region of the polygon of the main imagingobject that is perspectively projected to the texture images of the twoviewpoints decreases. Further, though not depicted, in a case where theshape of a polygon of a main imaging object is a recessed shape, whenthe opposite end portions between which a cavity is held in the depthdirections of the two viewpoints, then since the end portion in frontblocks the end portion in the back, the region of the polygon of themain imaging object to be perspectively projected to the texture imagesof the two viewpoints decreases.

Accordingly, the drawing section 302 generates texture images of twoviewpoints for a plurality of candidates for a pair of two viewpointsand selects a candidate for a pair of two viewpoints in which the regionof the rear face of the polygon to be perspectively projected isgreatest as an optimum pair.

(Description of First Determination Method of Candidate for Pair of TwoViewpoints)

FIG. 19 is a view illustrating a first determination method of acandidate for a pair of two viewpoints.

In the example of FIG. 19, a polygon of a main imaging object forms therobot 320 of FIG. 17. Further, the viewpoint O1 from between the currentpair of two viewpoints is on the left side of the left arm 321 on theaa′ cross section of FIG. 17, and the other viewpoint O2 is on the rightside of the right arm 323 on the aa′ cross section of FIG. 17. They aresimilar to those also in FIG. 20 hereinafter described.

As depicted in FIG. 19, in the first determination method, the viewpointcontrolling section 301 rotates a straight line 341, which interconnectsthe viewpoint O1 and the viewpoint O2 and passes the origin O,successively by a predetermined amount in one direction around theorigin O within a predetermined range to determine candidates for a pairof two viewpoints including the current pair of two viewpoints. Inparticular, in the first determination method, a candidate for a pair oftwo viewpoints moves on a circle centered at the origin O and having thestraight line 341 as a diameter.

In the example of FIG. 19, the viewpoint controlling section 301 rotatesthe straight line 341 within a range of a very small angle θrsuccessively by a predetermined amount in the clockwise direction or inthe counterclockwise direction in a direction parallel to the aa′ crosssection to determine candidates for a pair of two viewpoints includingthe current pair of two viewpoints.

(Description of Second Determination Method of Candidates for Pair ofTwo Viewpoints)

FIG. 20 is a view illustrating a second determination method ofcandidates for a pair of two viewpoints.

As depicted in FIG. 20, according to the second determination method,the viewpoint controlling section 301 determines candidates for a pairof two viewpoints including the current pair of two viewpoints byrotating the straight line 341 successively by a predetermined amount intwo or more directions within a predetermined range around the origin O.In particular, in the second determination method, candidates for a pairof two viewpoints move on a spherical face centered at the origin O andhaving the straight line 341 as a diameter.

In the example of FIG. 20, the viewpoint controlling section 301determines candidates for a pair of two viewpoints by rotating thestraight line 341 successively by a predetermined amount in theclockwise direction or in the counterclockwise direction within therange of the very small angle θr in a direction parallel to the aa′cross section. Further, the viewpoint controlling section 301 determinescandidates for a pair of two viewpoints by rotating the straight line341 successively by a predetermined amount in the clockwise direction orin the counterclockwise direction within the range of a very small angleφr in a direction perpendicular to the aa′ cross section.

As described above, in the first and second determination methods, theviewpoint controlling section 301 determines candidates for a pair oftwo viewpoints by rotating the straight line 341 interconnecting theviewpoint O1 and the viewpoint O2 at present within the range of a verysmall angle. Accordingly, the pair of two viewpoints varies stepwisesuch that the region of the rear face of a polygon to be perspectivelyprojected increases. Therefore, even in a case where a pair of twoviewpoints is to be determined in a unit of a frame, the pair of twoviewpoints does not vary by a great amount between frames. As a result,deterioration of the encoding efficiency of texture images and depthimages caused by a great variation of the pair of two viewpoints betweenframes can be prevented.

(Example of Table)

FIG. 21 is a view depicting an example of a table generated by theviewpoint controlling section 301 of FIG. 16.

As depicted in FIG. 21, into the table, viewpoint information and anazimuth angle and an elevation angle that indicate a sight linedirection are registered for each viewpoint.

The azimuth angle is an angle in an XZ plane direction defined by asight line direction and the Z axis, and an elevation angle is an angledefied by the sight line direction and the XZ plane. A line when avector in the positive direction of the Z axis extending from theviewpoint is horizontally rotated by an azimuth angle on the XZ planeand then rotated upwardly or downwardly in the Y axis direction by anelevation angle represents the sight line direction.

In the example of FIG. 21, the Z axis direction of the 3D modelcoordinate system is a direction heading from one to the other of thetwo viewpoints. Accordingly, the azimuth angle and the elevation angleat one viewpoint from between the two viewpoints are 0 degrees, and theviewpoint information is three-dimensional coordinates (0, 0, −1.0).Further, the azimuth angle of the other viewpoint is −180 degrees, andthe three-dimensional coordinates are (0, 0, 1.0).

Further, into the table, a rotation angle that is an angle in therotation direction of the perspective projection face around the sightline direction for each viewpoint is registered. Furthermore, into thetable, for each viewpoint, a transverse field angle that is a fieldangle in the transverse direction and a vertical field angle that is afield angle in the vertical direction as well as a transverse pixelnumber that is a pixel number in the transverse direction and a verticalpixel number that is a pixel number in the vertical direction of atexture image of the viewpoint are registered. In the example of FIG.21, both of the rotation angles of the two viewpoints are each 0degrees; the transverse field angles and the vertical field angles ofthe two viewpoints are each 90 degrees; and the transverse pixel numbersand the vertical pixel numbers of the two viewpoints are each 1024.

(Description of Processing of Image Processing Apparatus)

FIGS. 22 and 23 are flow charts illustrating a generation process of thegeneration apparatus 300 of FIG. 16. This generation process isperformed for each of frames of N picked up images and depth imagesacquired by the n imaging apparatus 11.

Processes at steps S51 to S54 of FIG. 22 are similar to the processes atsteps S11 to S14 of FIG. 11.

At step S55, the viewpoint controlling section 301 decides whether ornot the frame of the processing target is the top frame. In a case whereit is decided at step S55 that the frame of the processing target is notthe top frame, namely, in a case where a current pair of two viewpointsis set already, the processing advances to step S56.

At step S56, the viewpoint controlling section 301 decides on the basisof the generation frequency of candidates for a pair of two viewpoints,namely, on the basis of the update frequency of a pair of twoviewpoints, whether or not a plurality of candidates for a pair of twoviewpoints are to be determined. For example, in a case where thegeneration frequency of a candidate for a pair of two viewpoints is aunit of a frame, the viewpoint controlling section 301 decides that aplurality of candidates for a pair of two viewpoints are to bedetermined constantly. Alternatively, in a case where the generationfrequency of a candidate for a pair of two viewpoints is a unit of asequence or a unit of a GOP, when the frame of the processing target isthe top frame of a sequence or a GOP, the viewpoint controlling section301 decides to determine a plurality of candidates for a pair of twoviewpoints.

In a case where it is decided at step S56 that a plurality of candidatesfor a pair of two viewpoints are to be determined, the processingadvances to step S57. At step S57, the viewpoint controlling section 301determines a plurality of candidates for a pair of two viewpoints by thefirst or second determination method. The viewpoint controlling section301 generates and supplies viewpoint information of the plurality ofcandidates for a pair of two viewpoints to the drawing section 302.

At step S58, the drawing section 302 perspectively projects, for eachpiece of the viewpoint information, the rear face of each polygon to aperspective projection face with respect to each of the viewpoints ofcandidates for a pair of two viewpoints whose three-dimensionalcoordinates are indicated by the piece of viewpoint information togenerate texture images of the candidates for a pair of two viewpoints.

At step S59, the drawing section 302 selects, from among the candidatesfor a pair of two viewpoints, the candidate for a pair of two viewpointsin which the region of the rear face of a polygon to be perspectivelyprojected when a texture image is to be generated is greatest as optimumpair of two viewpoints. The drawing section 302 retains the viewpointinformation of the optimum pair of two viewpoints as viewpointinformation of the current pair of two viewpoints and supplies theviewpoint information to the viewpoint controlling section 301. Further,the viewpoint controlling section 301 determines texture images of theoptimum pair of two viewpoints as texture images of a final current pairof two viewpoints.

At step S60, the viewpoint controlling section 301 decides whether thefinal pair of two viewpoints corresponding to the viewpoint informationsupplied from the drawing section 302 is the current pair of twoviewpoints. In a case where it is decided at step S60 that the finalpair of two viewpoints is not the current pair of two viewpoints, theprocessing advances to step S61.

At step S61, the viewpoint controlling section 301 generates a tableincluding the viewpoint information supplied from the drawing section302 and supplies the table to the storage section 303 so as to bestored.

At step S62, the viewpoint controlling section 301 sets the optimum pairof two viewpoints to the current pair of two viewpoints, and advancesthe processing to step S65 in FIG. 23.

Conversely, in a case where it is decided at step S55 that the frame ofthe processing target is the top frame, namely, in a case where acurrent pair of two viewpoints is not set as yet, the processingadvances to step S63.

At step S63, the viewpoint controlling section 301 sets the current pairof two viewpoints to an initial value, and advances the processing tostep S64.

Conversely, in a case where it is decided at step S56 that a pluralityof candidates for a pair of two viewpoints are not to be determined, thedrawing section 302 perspectively projects the rear face of each polygonto the perspective projection face with respect to each of theviewpoints of the current pair of two viewpoints whose three-dimensionalcoordinates are represented by the retained viewpoint information togenerate texture images of the current pair of two viewpoints. Then, theprocessing advances to step S65 of FIG. 23.

Conversely, in a case where it is decided at step S60 that the optimumpair of two viewpoints is not the current pair of two viewpoints, sincethere is no necessity to change the current pair of two viewpoints, theprocessing skips steps S61 and S62 and advances to step S65 of FIG. 23.

At step S65 of FIG. 23, the drawing section 302 generates, on the basisof the polygons, depth images individually corresponding to the textureimages of the current pair of two viewpoints. The drawing section 302supplies the texture images and the depth images of the current pair oftwo viewpoints to the encoder 38.

Since processes at steps S66 to S69 are similar to the processes atsteps S17 to S20 of FIG. 11, description of them is omitted.

At step S70, the transmission section 304 reads out and transmits theviewpoint texture streams and the viewpoint depth streams of twoviewpoints, the omnidirectional texture stream and the omnidirectionaldepth stream, and the table from the storage section 39. Then, theprocessing ends.

In this manner, the generation apparatus 300 determines a pair of twoviewpoints such that texture images and depth images of two viewpointscan represent three-dimensional structures of a greater number ofregions. Accordingly, three-dimensional structures of a greater numberof regions can be represented using texture images and depth images oftwo viewpoints.

The configuration of the display apparatus in the second embodiment issame as the configuration of the display apparatus 200 of FIG. 12 exceptthat a table is used upon reconstruction by the reconstruction section204. In particular, since, in the second embodiment, the pair of twoviewpoints does not have fixed values determined in advance, thereconstruction section 204 generates, upon reconstruction,three-dimensional coordinates (X, Y, Z) of sampling points usingviewpoint information and sight line information of the viewpoints ofthe pair of two viewpoints registered in the table.

It is to be noted that, while, in the first and second embodiments, onlythe rear face of a polygon of a main imaging object is perspectivelyprojected to a texture image, also a front face of a polygon of abackground region may be perspectively projected to a texture image.

<Different Generation Method of Texture Image>

FIG. 24 is a view illustrating a generation method of a texture image inthis case.

In the example of FIG. 24, a sphere 81 of a main imaging object isformed.

In this case, in addition to the polygon that forms the sphere 81, apolygon of a background region 361 is generated at a position at whichthe position of the viewpoint O1 in the depth direction is on the backside farther than that of the sphere 81, and a polygon of a backgroundregion 362 is generated at a position at which the position of theviewpoint O2 in the depth direction is on the back side farther thanthat of the sphere 81. For the generation of the polygons of thebackground region 361 and the background region 362, N picked up imagesand depth images in the background region extracted by the regionextraction section 31 are used.

Further, for each of the viewpoints, the rear face of the polygonforming the sphere 81 and the front face of each of the polygons of thebackground region 361 and the background region 362 are perspectivelyprojected to generate a texture image.

As a result, as depicted in FIG. 24, on the texture image of theviewpoint O1, a region 153A of the rear face of the sphere 81 includedin the field angle 2θ of the viewpoint O1 and a region 361A that is notblocked by the region 153A from within a front face of a backgroundregion 361 are drawn. Further, on the texture image of the viewpoint O2,a region 153B of the rear face of the sphere 81 included in the fieldangle 2θ of the viewpoint O2 and a region 362A that is not blocked bythe region 153B from within a front face of the background region 362are drawn.

In a case where a texture image is generated in such a manner asdescribed above, the display apparatus 200 can generate a backgroundregion of a display image with high accuracy in comparison with analternative case in which only the rear face of a polygon of a mainimaging object is perspectively projected to the texture image. Further,also it is possible for the display apparatus 200 to delete thebackground region from a display image by drawing only three-dimensionalstructures existing within the field angle 2θ of both of the viewpointO1 and the viewpoint O2. It is to be noted that, in this case, thegeneration apparatus 12 (300) may not generate an omnidirectionaltexture stream and an omnidirectional depth stream.

<Different Example of Texture Image>

FIGS. 25A and 25B are views depicting a different example of a textureimage.

Although, in the foregoing description, the texture image is a textureimage of one viewpoint, it may be a composite of texture images of aviewpoint for the left eye and a viewpoint for the right eyecorresponding to the viewpoint.

In particular, as depicted in FIG. 25A, the texture image may be, forexample, a packed image 420 in which a texture image 421 of a viewpointfor the left eye and another texture image 422 of a viewpoint for theright eye that correspond to one viewpoint are packed in a transversedirection (horizontal direction).

Alternatively, as depicted in FIG. 25B, the texture image may be, forexample, a packed image 440 in which a texture image 421 and anothertexture image 422 are packed in a longitudinal direction (verticaldirection).

In a case where the texture image is a texture image in which images ofa viewpoint for the left eye and a viewpoint for the right eye arepacked as described above, the texture image obtained as a result ofdecoding is separated into the texture image of the viewpoint for theleft eye and the texture image of the viewpoint for the right eye. Then,a three-dimensional structure is generated for each eye.

Then, a display image for the left eye is generated from thethree-dimensional structure for the left eye on the basis of theviewpoint of the left eye corresponding to the viewpoint designated bythe viewer or the like, the sight line direction, and the field angle.Meanwhile, a display image for the right eye is generated from thethree-dimensional structure for the right eye on the basis of theviewpoint of the right eye corresponding to the viewpoint designated bythe viewer or the like, the sight line direction, and the field angle.

In the case where the display section 206 is 3D displayable, the displaysection 206 displays a display image for the left eye as an image forthe left eye and displays a display image for the right eye as an imagefor the right eye to 3D display a display image. On the other hand, inthe case where the display section 206 is not 3D displayable, thedisplay section 206 2D displays the display image for the left eye orthe display image for the right eye.

It is to be noted that, while the number of viewpoints is two in thefirst and second embodiments, the number of viewpoints is not limited totwo. Further, the two viewpoints may not be opposed to each other. Thesight line direction may be a direction from the viewpoint to a positionother than the origin.

Further, the generation apparatus 12 (300) may read out viewpointtexture streams and viewpoint depth streams of two viewpoints as well asan omnidirectional stream and an omnidirectional depth stream stored inthe storage section 39 (303) and transmit them to the display apparatus200 only when a request is received from the display apparatus 200. Thissimilarly applies also to transmission of the table.

Furthermore, also in the first embodiment, the generation apparatus 12may generate a table including viewpoint information of two viewpointsdetermined in advance and transmit the table to the display apparatus200 similarly as in the second embodiment.

Third Embodiment

The configuration of a third embodiment of a distribution system towhich the present disclosure is applied is same as the configurationdescribed above, for example, of the generation apparatus 12 of FIG. 1or the display apparatus 200 depicted in FIG. 12 except that tan axisprojection (details are hereinafter described) is performed in place ofperspective projection. Accordingly, in the following, description isgiven only of tan axis projection.

(Description of Coordinate System of Projection Face)

FIG. 26 is a view illustrating a coordinate system of a projection face.

It is to be noted that, in the third embodiment, the projection face isa two-dimensional plane to which, when the generation apparatus 12generates a high resolution image, an omnidirectional image mapped to asphere is tan axis projected or a viewing range in which, when thedisplay apparatus 200 generates a display image, a 3D model image is tanaxis projected.

In the example of FIG. 26, a projection face 451 whose z is −1.0 is setin a three-dimensional xyz coordinate system of a 3D model. In thiscase, a two-dimensional st coordinate system in which the center O′ ofthe projection face 451 is the origin and the horizontal direction ofthe projection face 451 is an s direction while the vertical directionis a t direction is a coordinate system of the projection face 451.

It is to be noted that, in the following description, a vector 452heading from the origin O of the xyz coordinate system to coordinates(s, t) of the st coordinate system is referred to as vector (s, t, −1.0)using the coordinates (s, t) and −1.0 that is the distance from theorigin O to the projection face 451.

(Description of Tan Axis Projection)

FIG. 27 is a view illustrating tan axis projection (tangential axisprojection).

FIG. 27 is a view of the projection face 451 as viewed in the negativedirection of z. In the example of FIG. 27, in the st coordinate system,the minimum values of the s value and the t value of the projection face451 are −1.0, and the maximum values are 1.0.

In this case, in perspective projection, a projection point is set onthe projection face 451 such that the projection vector heading from theorigin O to the projection point on the projection face 451 becomes avector (s′, t′, −1.0). It is to be noted that s′ indicates a value ateach of predetermined distances provided within a range of the s valuefrom −1.0 to 1.0, and t′ indicates a value at each of predetermineddistances provided within a range of the t value from −1.0 to 1.0.Accordingly, projection points in perspective projection are uniform onthe projection face 451.

In contrast, if the field angle of the projection face 451 is θw (in theexample of FIG. 27, π/2), then in tan axis projection, a projectionpoint is set on the projection face 451 such that the projection vectoris a vector (tan(s′*w/2), tan(t′*θw/2), −1.0).

In particular, if s′*θw/2 is represented by θ and t′*θw/2 is representedby φ, then the vector (tan(s′*θw/2), tan(t′*θw/2), −1.0) becomes avector (tan θ, tan φ, −1.0). At this time, if the field angle θw comesclose to π, then tan θ and tan φ diverge to infinity. Accordingly, thevector (tan θ, tan φ, −1.0) is corrected to a vector (sin θ*cos φ, cosθ*sin φ, −cos θ*cos φ) such that tan θ and tan φ do not diverge toinfinity, and a projection point is set on the projection face 451 suchthat the projection vector becomes the vector (sin θ*cos φ, cos θ*sin φ,−cos θ*cos φ). Accordingly, in tan axis projection, angles defined byevery projection vectors corresponding to projection points neighboringwith each other become equal.

It is to be noted that, similarly to a logarithmic axis (log scale),tan(s′*θw/2) and tan(t′*θw/2) are grasped as s′ and t′ of the tan axis.Accordingly, in the present specification, projection where theprojection vector is the vector (tan(s′*θw/2), tan(t′*θw/2), −1.0) isreferred to as tan axis projection.

Fourth Embodiment

(Description of Computer to which Present Disclosure is Applied)

While the series of processes described above can be executed byhardware, it may otherwise be executed by software. In a case where theseries of processes is executed by software, a program that constitutesthe software is installed into a computer. Here, the computer includes acomputer incorporated in hardware for exclusive use, for example, apersonal computer for universal use that can execute various functionsby installing various programs, and the like.

FIG. 28 is a block diagram depicting a configuration example of hardwareof a computer that executes the series of processes describedhereinabove in accordance with a program.

In the computer 500, a CPU (Central Processing Unit) 501, a ROM (ReadOnly Memory) 502, and a RAM (Random Access Memory) 503 are connected toeach other by a bus 504.

To the bus 504, an input/output interface 505 is connected further. Tothe input/output interface 505, an inputting section 506, an outputtingsection 507, a storage section 508, a communication section 509, and adrive 510 are connected.

The inputting section 506 includes a keyboard, a mouse, a microphone,and the like. The outputting section 507 includes a display, a speaker,and the like. The storage section 508 includes a hard disk, anonvolatile memory, and the like. The communication section 509 includesa network interface or the like. The drive 510 drives a removable medium511 such as a magnetic disk, an optical disk, a magneto-optical disk, asemiconductor memory, or the like.

In the computer 500 configured in such a manner as described above, theCPU 501 loads a program stored, for example, in the storage section 508into the RAM 503 through the input/output interface 505 and the bus 504to perform the series of processes described above.

The program that is executed by the computer 500 (CPU 501) can berecorded into and provided as the removable medium 511, for example, asa package medium or the like. Further, the program can be providedthrough a wired or wireless transmission medium such as a local areanetwork, the Internet, a digital satellite broadcast, or the like.

In the computer 500, a program can be installed into the storage section508 through the input/output interface 505 by mounting a removablemedium 511 on the drive 510. Further, the program can be received by thecommunication section 509 through a wired or wireless transmissionmedium and installed into the storage section 508. Further, the programcan be installed in advance into the ROM 502 or the storage section 508.

It is to be noted that the program executed by the computer 500 may be aprogram in which processes are performed in time series in accordancewith the order described in the present specification or may be aprogram in which processes are executed in parallel or at a necessarytiming such as, for example, when the program is called, or the like.

Application Example

The technology according to the present disclosure can be applied tovarious products. For example, the technology according to the presentdisclosure may be implemented as an apparatus that is incorporated invarious types of mobile bodies such as automobiles, electric cars,hybrid electric cars, motorcycles, bicycles, personal mobilities,airplanes, drones, ships, robots, construction machines, agriculturalmachines (tractors), and the like.

FIG. 29 is a block diagram depicting an example of schematicconfiguration of a vehicle control system 7000 as an example of a mobilebody control system to which the technology according to an embodimentof the present disclosure can be applied. The vehicle control system7000 includes a plurality of electronic control units connected to eachother via a communication network 7010. In the example depicted in FIG.29, the vehicle control system 7000 includes a driving system controlunit 7100, a body system control unit 7200, a battery control unit 7300,an outside-vehicle information detecting unit 7400, an in-vehicleinformation detecting unit 7500, and an integrated control unit 7600.The communication network 7010 connecting the plurality of control unitsto each other may, for example, be a vehicle-mounted communicationnetwork compliant with an arbitrary standard such as controller areanetwork (CAN), local interconnect network (LIN), local area network(LAN), FlexRay, or the like.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various kinds of programs; a storagesection that stores the programs executed by the microcomputer,parameters used for various kinds of operations, or the like; and adriving circuit that drives various kinds of control target devices.Each of the control units further includes: a network interface (I/F)for performing communication with other control units via thecommunication network 7010; and a communication I/F for performingcommunication with a device, a sensor, or the like within and withoutthe vehicle by wire communication or radio communication. A functionalconfiguration of the integrated control unit 7600 illustrated in FIG. 29includes a microcomputer 7610, a general-purpose communication I/F 7620,a dedicated communication I/F 7630, a positioning section 7640, a beaconreceiving section 7650, an in-vehicle device I/F 7660, a sound/imageoutput section 7670, a vehicle-mounted network I/F 7680, and a storagesection 7690. The other control units similarly include a microcomputer,a communication I/F, a storage section, and the like.

The driving system control unit 7100 controls the operation of devicesrelated to the driving system of the vehicle in accordance with variouskinds of programs. For example, the driving system control unit 7100functions as a control device for a driving force generating device forgenerating the driving force of the vehicle, such as an internalcombustion engine, a driving motor, or the like, a driving forcetransmitting mechanism for transmitting the driving force to wheels, asteering mechanism for adjusting the steering angle of the vehicle, abraking device for generating the braking force of the vehicle, and thelike. The driving system control unit 7100 may have a function as acontrol device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The driving system control unit 7100 is connected with a vehicle statedetecting section 7110. The vehicle state detecting section 7110, forexample, includes at least one of a gyro sensor that detects the angularvelocity of axial rotational movement of a vehicle body, an accelerationsensor that detects the acceleration of the vehicle, and sensors fordetecting an amount of operation of an accelerator pedal, an amount ofoperation of a brake pedal, the steering angle of a steering wheel, anengine speed or the rotational speed of wheels, and the like. Thedriving system control unit 7100 performs arithmetic processing using asignal input from the vehicle state detecting section 7110, and controlsthe internal combustion engine, the driving motor, an electric powersteering device, the brake device, and the like.

The body system control unit 7200 controls the operation of variouskinds of devices provided to the vehicle body in accordance with variouskinds of programs. For example, the body system control unit 7200functions as a control device for a keyless entry system, a smart keysystem, a power window device, or various kinds of lamps such as aheadlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or thelike. In this case, radio waves transmitted from a mobile device as analternative to a key or signals of various kinds of switches can beinput to the body system control unit 7200. The body system control unit7200 receives these input radio waves or signals, and controls a doorlock device, the power window device, the lamps, or the like of thevehicle.

The battery control unit 7300 controls a secondary battery 7310, whichis a power supply source for the driving motor, in accordance withvarious kinds of programs. For example, the battery control unit 7300 issupplied with information about a battery temperature, a battery outputvoltage, an amount of charge remaining in the battery, or the like froma battery device including the secondary battery 7310. The batterycontrol unit 7300 performs arithmetic processing using these signals,and performs control for regulating the temperature of the secondarybattery 7310 or controls a cooling device provided to the battery deviceor the like.

The outside-vehicle information detecting unit 7400 detects informationabout the outside of the vehicle including the vehicle control system7000. For example, the outside-vehicle information detecting unit 7400is connected with at least one of an imaging section 7410 and anoutside-vehicle information detecting section 7420. The imaging section7410 includes at least one of a time-of-flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Theoutside-vehicle information detecting section 7420, for example,includes at least one of an environmental sensor for detecting currentatmospheric conditions or weather conditions and a peripheralinformation detecting sensor for detecting another vehicle, an obstacle,a pedestrian, or the like on the periphery of the vehicle including thevehicle control system 7000.

The environmental sensor, for example, may be at least one of a raindrop sensor detecting rain, a fog sensor detecting a fog, a sunshinesensor detecting a degree of sunshine, and a snow sensor detecting asnowfall. The peripheral information detecting sensor may be at leastone of an ultrasonic sensor, a radar device, and a LIDAR device (Lightdetection and Ranging device, or Laser imaging detection and rangingdevice). Each of the imaging section 7410 and the outside-vehicleinformation detecting section 7420 may be provided as an independentsensor or device, or may be provided as a device in which a plurality ofsensors or devices are integrated.

FIG. 30 depicts an example of installation positions of the imagingsection 7410 and the outside-vehicle information detecting section 7420.Imaging sections 7910, 7912, 7914, 7916, and 7918 are, for example,disposed at at least one of positions on a front nose, sideview mirrors,a rear bumper, and a back door of the vehicle 7900 and a position on anupper portion of a windshield within the interior of the vehicle. Theimaging section 7910 provided to the front nose and the imaging section7918 provided to the upper portion of the windshield within the interiorof the vehicle obtain mainly an image of the front of the vehicle 7900.The imaging sections 7912 and 7914 provided to the sideview mirrorsobtain mainly an image of the sides of the vehicle 7900. The imagingsection 7916 provided to the rear bumper or the back door obtains mainlyan image of the rear of the vehicle 7900. The imaging section 7918provided to the upper portion of the windshield within the interior ofthe vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.

Incidentally, FIG. 30 depicts an example of photographing ranges of therespective imaging sections 7910, 7912, 7914, and 7916. An imaging rangea represents the imaging range of the imaging section 7910 provided tothe front nose. Imaging ranges b and c respectively represent theimaging ranges of the imaging sections 7912 and 7914 provided to thesideview mirrors. An imaging range d represents the imaging range of theimaging section 7916 provided to the rear bumper or the back door. Abird's-eye image of the vehicle 7900 as viewed from above can beobtained by superimposing image data imaged by the imaging sections7910, 7912, 7914, and 7916, for example.

Outside-vehicle information detecting sections 7920, 7922, 7924, 7926,7928, and 7930 provided to the front, rear, sides, and corners of thevehicle 7900 and the upper portion of the windshield within the interiorof the vehicle may be, for example, an ultrasonic sensor or a radardevice. The outside-vehicle information detecting sections 7920, 7926,and 7930 provided to the front nose of the vehicle 7900, the rearbumper, the back door of the vehicle 7900, and the upper portion of thewindshield within the interior of the vehicle may be a LIDAR device, forexample. These outside-vehicle information detecting sections 7920 to7930 are used mainly to detect a preceding vehicle, a pedestrian, anobstacle, or the like.

Returning to FIG. 29, the description will be continued. Theoutside-vehicle information detecting unit 7400 makes the imagingsection 7410 image an image of the outside of the vehicle, and receivesimaged image data. In addition, the outside-vehicle informationdetecting unit 7400 receives detection information from theoutside-vehicle information detecting section 7420 connected to theoutside-vehicle information detecting unit 7400. In a case where theoutside-vehicle information detecting section 7420 is an ultrasonicsensor, a radar device, or a LIDAR device, the outside-vehicleinformation detecting unit 7400 transmits an ultrasonic wave, anelectromagnetic wave, or the like, and receives information of areceived reflected wave. On the basis of the received information, theoutside-vehicle information detecting unit 7400 may perform processingof detecting an object such as a human, a vehicle, an obstacle, a sign,a character on a road surface, or the like, or processing of detecting adistance thereto. The outside-vehicle information detecting unit 7400may perform environment recognition processing of recognizing arainfall, a fog, road surface conditions, or the like on the basis ofthe received information. The outside-vehicle information detecting unit7400 may calculate a distance to an object outside the vehicle on thebasis of the received information.

In addition, on the basis of the received image data, theoutside-vehicle information detecting unit 7400 may perform imagerecognition processing of recognizing a human, a vehicle, an obstacle, asign, a character on a road surface, or the like, or processing ofdetecting a distance thereto. The outside-vehicle information detectingunit 7400 may subject the received image data to processing such asdistortion correction, alignment, or the like, and combine the imagedata imaged by a plurality of different imaging sections 7410 togenerate a bird's-eye image or a panoramic image. The outside-vehicleinformation detecting unit 7400 may perform viewpoint conversionprocessing using the image data imaged by the imaging section 7410including the different imaging parts.

The in-vehicle information detecting unit 7500 detects information aboutthe inside of the vehicle. The in-vehicle information detecting unit7500 is, for example, connected with a driver state detecting section7510 that detects the state of a driver. The driver state detectingsection 7510 may include a camera that images the driver, a biosensorthat detects biological information of the driver, a microphone thatcollects sound within the interior of the vehicle, or the like. Thebiosensor is, for example, disposed in a seat surface, the steeringwheel, or the like, and detects biological information of an occupantsitting in a seat or the driver holding the steering wheel. On the basisof detection information input from the driver state detecting section7510, the in-vehicle information detecting unit 7500 may calculate adegree of fatigue of the driver or a degree of concentration of thedriver, or may determine whether the driver is dozing. The in-vehicleinformation detecting unit 7500 may subject an audio signal obtained bythe collection of the sound to processing such as noise cancelingprocessing or the like.

The integrated control unit 7600 controls general operation within thevehicle control system 7000 in accordance with various kinds ofprograms. The integrated control unit 7600 is connected with an inputsection 7800. The input section 7800 is implemented by a device capableof input operation by an occupant, such, for example, as a touch panel,a button, a microphone, a switch, a lever, or the like. The integratedcontrol unit 7600 may be supplied with data obtained by voicerecognition of voice input through the microphone. The input section7800 may, for example, be a remote control device using infrared rays orother radio waves, or an external connecting device such as a mobiletelephone, a personal digital assistant (PDA), or the like that supportsoperation of the vehicle control system 7000. The input section 7800 maybe, for example, a camera. In that case, an occupant can inputinformation by gesture. Alternatively, data may be input which isobtained by detecting the movement of a wearable device that an occupantwears. Further, the input section 7800 may, for example, include aninput control circuit or the like that generates an input signal on thebasis of information input by an occupant or the like using theabove-described input section 7800, and which outputs the generatedinput signal to the integrated control unit 7600. An occupant or thelike inputs various kinds of data or gives an instruction for processingoperation to the vehicle control system 7000 by operating the inputsection 7800.

The storage section 7690 may include a read only memory (ROM) thatstores various kinds of programs executed by the microcomputer and arandom access memory (RAM) that stores various kinds of parameters,operation results, sensor values, or the like. In addition, the storagesection 7690 may be implemented by a magnetic storage device such as ahard disc drive (HDD) or the like, a semiconductor storage device, anoptical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a communication I/F usedwidely, which communication I/F mediates communication with variousapparatuses present in an external environment 7750. The general-purposecommunication I/F 7620 may implement a cellular communication protocolsuch as global system for mobile communications (GSM), worldwideinteroperability for microwave access (WiMAX), long term evolution(LTE)), LTE-advanced (LTE-A), or the like, or another wirelesscommunication protocol such as wireless LAN (referred to also aswireless fidelity (Wi-Fi), Bluetooth, or the like. The general-purposecommunication I/F 7620 may, for example, connect to an apparatus (forexample, an application server or a control server) present on anexternal network (for example, the Internet, a cloud network, or acompany-specific network) via a base station or an access point. Inaddition, the general-purpose communication I/F 7620 may connect to aterminal present in the vicinity of the vehicle (which terminal is, forexample, a terminal of the driver, a pedestrian, or a store, or amachine type communication (MTC) terminal) using a peer to peer (P2P)technology, for example.

The dedicated communication I/F 7630 is a communication I/F thatsupports a communication protocol developed for use in vehicles. Thededicated communication I/F 7630 may implement a standard protocol such,for example, as wireless access in vehicle environment (WAVE), which isa combination of institute of electrical and electronic engineers (IEEE)802.11p as a lower layer and IEEE 1609 as a higher layer, dedicatedshort range communications (DSRC), or a cellular communication protocol.The dedicated communication I/F 7630 typically carries out V2Xcommunication as a concept including one or more of communicationbetween a vehicle and a vehicle (Vehicle to Vehicle), communicationbetween a road and a vehicle (Vehicle to Infrastructure), communicationbetween a vehicle and a home (Vehicle to Home), and communicationbetween a pedestrian and a vehicle (Vehicle to Pedestrian).

The positioning section 7640, for example, performs positioning byreceiving a global navigation satellite system (GNSS) signal from a GNSSsatellite (for example, a GPS signal from a global positioning system(GPS) satellite), and generates positional information including thelatitude, longitude, and altitude of the vehicle. Incidentally, thepositioning section 7640 may identify a current position by exchangingsignals with a wireless access point, or may obtain the positionalinformation from a terminal such as a mobile telephone, a personalhandyphone system (PHS), or a smart phone that has a positioningfunction.

The beacon receiving section 7650, for example, receives a radio wave oran electromagnetic wave transmitted from a radio station installed on aroad or the like, and thereby obtains information about the currentposition, congestion, a closed road, a necessary time, or the like.Incidentally, the function of the beacon receiving section 7650 may beincluded in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 present within the vehicle. The in-vehicledevice I/F 7660 may establish wireless connection using a wirelesscommunication protocol such as wireless LAN, Bluetooth, near fieldcommunication (NFC), or wireless universal serial bus (WUSB). Inaddition, the in-vehicle device I/F 7660 may establish wired connectionby universal serial bus (USB), high-definition multimedia interface(HDMI), mobile high-definition link (MHL), or the like via a connectionterminal (and a cable if necessary) not depicted in the figures. Thein-vehicle devices 7760 may, for example, include at least one of amobile device and a wearable device possessed by an occupant and aninformation device carried into or attached to the vehicle. Thein-vehicle devices 7760 may also include a navigation device thatsearches for a path to an arbitrary destination. The in-vehicle deviceI/F 7660 exchanges control signals or data signals with these in-vehicledevices 7760.

The vehicle-mounted network I/F 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The vehicle-mounted network I/F 7680 transmits andreceives signals or the like in conformity with a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 controls thevehicle control system 7000 in accordance with various kinds of programson the basis of information obtained via at least one of thegeneral-purpose communication I/F 7620, the dedicated communication I/F7630, the positioning section 7640, the beacon receiving section 7650,the in-vehicle device I/F 7660, and the vehicle-mounted network I/F7680. For example, the microcomputer 7610 may calculate a control targetvalue for the driving force generating device, the steering mechanism,or the braking device on the basis of the obtained information about theinside and outside of the vehicle, and output a control command to thedriving system control unit 7100. For example, the microcomputer 7610may perform cooperative control intended to implement functions of anadvanced driver assistance system (ADAS) which functions includecollision avoidance or shock mitigation for the vehicle, followingdriving based on a following distance, vehicle speed maintainingdriving, a warning of collision of the vehicle, a warning of deviationof the vehicle from a lane, or the like. In addition, the microcomputer7610 may perform cooperative control intended for automatic driving,which makes the vehicle to travel autonomously without depending on theoperation of the driver, or the like, by controlling the driving forcegenerating device, the steering mechanism, the braking device, or thelike on the basis of the obtained information about the surroundings ofthe vehicle.

The microcomputer 7610 may generate three-dimensional distanceinformation between the vehicle and an object such as a surroundingstructure, a person, or the like, and generate local map informationincluding information about the surroundings of the current position ofthe vehicle, on the basis of information obtained via at least one ofthe general-purpose communication I/F 7620, the dedicated communicationI/F 7630, the positioning section 7640, the beacon receiving section7650, the in-vehicle device I/F 7660, and the vehicle-mounted networkI/F 7680. In addition, the microcomputer 7610 may predict danger such ascollision of the vehicle, approaching of a pedestrian or the like, anentry to a closed road, or the like on the basis of the obtainedinformation, and generate a warning signal. The warning signal may, forexample, be a signal for producing a warning sound or lighting a warninglamp.

The sound/image output section 7670 transmits an output signal of atleast one of a sound and an image to an output device capable ofvisually or auditorily notifying information to an occupant of thevehicle or the outside of the vehicle. In the example of FIG. 29, anaudio speaker 7710, a display section 7720, and an instrument panel 7730are illustrated as the output device. The display section 7720 may, forexample, include at least one of an on-board display and a head-updisplay. The display section 7720 may have an augmented reality (AR)display function. The output device may be other than these devices, andmay be another device such as headphones, a wearable device such as aneyeglass type display worn by an occupant or the like, a projector, alamp, or the like. In a case where the output device is a displaydevice, the display device visually displays results obtained by variouskinds of processing performed by the microcomputer 7610 or informationreceived from another control unit in various forms such as text, animage, a table, a graph, or the like. In addition, in a case where theoutput device is an audio output device, the audio output deviceconverts an audio signal constituted of reproduced audio data or sounddata or the like into an analog signal, and auditorily outputs theanalog signal.

Incidentally, at least two control units connected to each other via thecommunication network 7010 in the example depicted in FIG. 29 may beintegrated into one control unit. Alternatively, each individual controlunit may include a plurality of control units. Further, the vehiclecontrol system 7000 may include another control unit not depicted in thefigures. In addition, part or the whole of the functions performed byone of the control units in the above description may be assigned toanother control unit. That is, predetermined arithmetic processing maybe performed by any of the control units as long as information istransmitted and received via the communication network 7010. Similarly,a sensor or a device connected to one of the control units may beconnected to another control unit, and a plurality of control units maymutually transmit and receive detection information via thecommunication network 7010.

It is to be noted that a computer program for implementing the functionsof the generation apparatus 12 (300) and the display apparatus 200according to the present embodiments described hereinabove withreference to FIGS. 1, 2, 3A, 3B, 3C, 4A, 4B, 5, 6, 7, 8A, 8B, 8C, 9A,9B, 10A, 10B, 11, 12, 13, 14, 15, 16, 17, 18A, 18B, 19, 20, 21, 22, 23,24, 25A, 25B, 26, and 27 can be incorporated into some control unit orthe like. Further, also it is possible to provide a computer-readablerecording medium in which such a computer program as just described isstored. The recording medium may be, for example, a magnetic disk, anoptical disk, a magneto-optical disk, a flash memory, or the like.Further, the computer program described above may be distributed, forexample, through a network without using a recording medium.

In the vehicle control system 7000 described above, the generationapparatus 12 (300) and the display apparatus 200 according to thepresent embodiments described hereinabove with reference to FIGS. 1, 2,3A, 3B, 3C, 4A, 4B, 5, 6, 7, 8A, 8B, 8C, 9A, 9B, 10A, 10B, 11, 12, 13,14, 15, 16, 17, 18A, 18B, 19, 20, 21, 22, 23, 24, 25A, 25B, 26, and 27can be applied. In this case, for example, the generation apparatus 12(300) and the display apparatus 200 are integrated and correspond to themicrocomputer 7610, the storage section 7690, and the display section7720. Further, the imaging apparatus 11 corresponds to the imagingsection 7410. In this case, for example, the vehicle control system 7000can represent a three-dimensional structure of a greater number ofregions using texture images and depth images of two viewpoints.

Further, at least part of the components of the generation apparatus 12(300) and the display apparatus 200 described hereinabove with referenceto FIGS. 1, 2, 3A, 3B, 3C, 4A, 4B, 5, 6, 7, 8A, 8B, 8C, 9A, 9B, 10A,10B, 11, 12, 13, 14, 15, 16, 17, 18A, 18B, 19, 20, 21, 22, 23, 24, 25A,25B, 26, and 27 may be implemented by a module for the vehicle controlsystem 7000 depicted in FIG. 29 (for example, by an integrated circuitmodule configured by one die). As an alternative, the generationapparatus 12 (300) and the display apparatus 200 described withreference to FIGS. 1, 2, 3A, 3B, 3C, 4A, 4B, 5, 6, 7, 8A, 8B, 8C, 9A,9B, 10A, 10B, 11, 12, 13, 14, 15, 16, 17, 18A, 18B, 19, 20, 21, 22, 23,24, 25A, 25B, 26, and 27 may be implemented by a plurality of controlunits of the vehicle control system 7000 depicted in FIG. 29.

It is to be noted that the advantageous effects described herein areillustrative only and are not restrictive, and other advantages may beavailable.

Further, the embodiment of the present disclosure is not limited to theembodiments described hereinabove, and various alterations are possiblewithout departing from the subject matter of the present disclosure.

For example, the present disclosure can assume a configuration for cloudcomputing in which one function is shared by a plurality of apparatusesthrough a network and processed in collaboration.

Further, the steps described hereinabove in connection with the flowcharts can be executed by a single apparatus or can be executed bysharing by a plurality of apparatuses.

Furthermore, in a case where one step includes a plurality of processes,the plurality of processes included in the one step can be executed by asingle apparatus and also can be executed by sharing by a plurality ofapparatuses.

It is to be noted that the present disclosure can assume such aconfiguration as described below.

(1)

An image processing apparatus, including:

an image generation section configured to generate a texture image of apredetermined viewpoint using a texture image obtained by projecting, toa projection face perpendicular to a sight line direction heading fromeach of two viewpoints which are opposed to each other across the centerof a polygon, toward a center of the polygon, a rear face of the polygonand a depth image corresponding to the texture image of each of theviewpoints.

(2)

The image processing apparatus according to (1), in which

the image generation section generates a texture image of thepredetermined viewpoint on a basis of viewpoint information indicativeof positions of the two viewpoints.

(3)

The image processing apparatus according to (1) or (2), in which

the polygon is generated using picked up images acquired by a pluralityof imaging apparatuses which are disposed around an imaging objectcorresponding to the polygon and include at least part of the imagingobject in an imaging range thereof, and depth images corresponding tothe picked up images.

(4)

The image processing apparatus according to (3), in which

the image generation section generates a texture image of thepredetermined viewpoint using a texture image of an omnidirectionalimage generated using the picked up images acquired by the plurality ofimaging apparatuses and a depth image of the omnidirectional imagegenerated using the depth images corresponding to the picked up imagesacquired by the plurality of imaging apparatuses.

(5)

An image processing method executed by an image processing apparatus,including:

an image generation step of generating a texture image of apredetermined viewpoint using a texture image obtained by projecting, toa projection face perpendicular to a sight line direction heading fromeach of two viewpoints which are opposed to each other across a centerof a polygon, toward the center of the polygon, a rear face of thepolygon and a depth image corresponding to the texture image of each ofthe viewpoints.

(6)

An image processing apparatus, including:

an image generation section configured to generate a texture image byprojecting, to a projection face perpendicular to a sight line directionheading from each of two viewpoints which are opposed to each otheracross a center of a polygon, toward the center of the polygon, a rearface of the polygon and generate a depth image corresponding to thetexture image of each of the viewpoints.

(7)

The image processing apparatus according to (6), further including:

a viewpoint information generation section configured to generateviewpoint information indicative of positions of the two viewpoints.

(8)

The image processing apparatus according to (7), in which

the viewpoint information generation section generates a plurality ofpieces of the viewpoint information, and

the image generation section generates, for each of the pieces of theviewpoint information generated by the viewpoint information generationsection, texture images of the two viewpoints, sets the texture imagesof the two viewpoints corresponding to the piece of the viewpointinformation in which the region of the rear face of the polygonprojected when the texture images of the two viewpoints are generatedfrom among the plurality of pieces of the viewpoint information as finaltexture images of the two viewpoints, and generates depth imagesindividually corresponding to the final texture images of the twoviewpoints.

(9)

The image processing apparatus according to (8), in which

the two viewpoints are determined by successively rotating a straightline which interconnects predetermined two viewpoints and passes acenter of the polygon, by a predetermined amount in at least onedirection around the center of the polygon.

(10)

The image processing apparatus according to (8), in which

the viewpoint information generation section outputs pieces of viewpointinformation individually corresponding to the final texture images ofthe two viewpoints.

(11)

The image processing apparatus according to any of (6) to (10), furtherincluding:

a polygon generation section configured to generate the polygon usingpicked up images acquired by a plurality of imaging apparatuses whichare disposed around an imaging object corresponding to the polygon andinclude at least part of the imaging object in an imaging range thereof,and depth images corresponding to the picked up images.

(12)

The image processing apparatus according to (11), further including:

an omnidirectional image generation section configured to generate atexture image of an omnidirectional image using the picked up imagesacquired by the plurality of image pickup apparatuses and generate adepth image of the omnidirectional image using depth imagescorresponding to the picked up images acquired by the plurality ofimaging apparatuses.

(13)

An image processing method executed by an image processing apparatus,including:

an image generation step of generating a texture image by projecting, toa projection face perpendicular to a sight line direction heading fromeach of two viewpoints which are opposed to each other across a centerof a polygon, toward the center of the polygon, a rear face of thepolygon and generating a depth image corresponding to the texture imageof each of the viewpoints.

REFERENCE SIGNS LIST

11-1 to 11-N imaging apparatus, 12 generation apparatus, 34 polygongeneration section, 35 drawing section, 61 imaging object, 81 sphere,200 display apparatus, 205 drawing section, 300 generation apparatus,301 viewpoint controlling section, 302 drawing section, 341 straightline

The invention claimed is:
 1. An image processing apparatus, comprising:a central processing unit (CPU) configured to generate a first textureimage of a specific viewpoint, based on each of a second texture imageof a first viewpoint, a third texture image of a second viewpoint, afirst depth image corresponding to the second texture image, and asecond depth image corresponding to the third texture image, wherein thesecond texture image is obtained based on projection of a first rearface of a polygon to a first projection face with respect to the firstviewpoint, the third texture image is obtained based on projection of asecond rear face of the polygon to a second projection face with respectto the second viewpoint, the first projection face is perpendicular to afirst sight line direction, the second projection face is perpendicularto a second sight line direction, the first sight line direction is fromthe first viewpoint to a center of the polygon, the second sight linedirection is from the second viewpoint to the center of the polygon, thefirst viewpoint and the second viewpoint are opposed to each otheracross the center of the polygon; the second texture image correspondsto a first reconstructed portion of the polygon, the third texture imagecorresponds to a second reconstructed portion of the polygon, and eachof the first reconstructed portion of the polygon and the secondreconstructed portion of the polygon is greater than one-half of thepolygon.
 2. The image processing apparatus according to claim 1, whereinthe CPU is further configured to generate a fourth texture image of thespecific viewpoint based on viewpoint information indicative ofpositions of the first viewpoint and the second viewpoint.
 3. The imageprocessing apparatus according to claim 1, wherein the polygon isgenerated based on picked up images and depth images corresponding tothe picked up images, the picked up images are acquired by a pluralityof cameras which is around an imaging object, the imaging objectcorresponds to the polygon, and at least a part of the imaging object isin an imaging range of each of the plurality of cameras.
 4. The imageprocessing apparatus according to claim 3, wherein the CPU is furtherconfigured to generate a fourth texture image of the specific viewpointbased on a fifth texture image of an omnidirectional image and a thirddepth image of the omnidirectional image, the fifth texture image of theomnidirectional image is based on the picked up images, and the thirddepth image of the omnidirectional image is based on the depth imagescorresponding to the picked up images.
 5. An image processing method,comprising: in an image processing apparatus: generating a first textureimage of a specific viewpoint, based on each of a second texture imageof a first viewpoint, a third texture image of a second viewpoint, afirst depth image corresponding to the second texture image, and asecond depth image corresponding to the third texture image, wherein thesecond texture image is obtained based on projection of a first rearface of a polygon to a first projection face with respect to the firstviewpoint, the third texture image is obtained based on projection of asecond rear face of the polygon to a second projection face with respectto the second viewpoint, the first projection face is perpendicular to afirst sight line direction, the second projection face is perpendicularto a second sight line direction, the first sight line direction is fromthe first viewpoint to a center of the polygon, the second sight linedirection is from the second viewpoint to the center of the polygon, thefirst viewpoint and the second viewpoint are opposed to each otheracross the center of the polygon, the second texture image correspondsto a first reconstructed portion of the polygon, the third texture imagecorresponds to a second reconstructed portion of the polygon, and eachof the first reconstructed portion of the polygon and the secondreconstructed portion of the polygon is greater than one-half of thepolygon.
 6. An image processing apparatus, comprising: a centralprocessing unit (CPU) configured to: generate each of a first textureimage of a first viewpoint and a second texture image of a secondviewpoint, wherein the generation of the first texture image is based onprojection of a first rear face of a polygon to a first projection facewith respect to the first viewpoint, the generation of the secondtexture image is based on projection of a second rear face of thepolygon to a second projection face with respect to the secondviewpoint, the first projection face is perpendicular to a first sightline direction, the second projection face is perpendicular to a secondsight line direction, the first sight line direction is from the firstviewpoint to a center of the polygon, the second sight line direction isfrom the second viewpoint to the center of the polygon, the firstviewpoint and the second viewpoint are opposed to each other across thecenter of the polygon, the first texture image corresponds to a firstreconstructed portion of the polygon, the second texture imagecorresponds to a second reconstructed portion of the polygon, and eachof the first reconstructed portion of the polygon and the secondreconstructed portion of the polygon is greater than one-half of thepolygon; and generate a first depth image and a second depth image,wherein the first depth image corresponds to the first texture image andthe second depth image corresponds to the second texture image.
 7. Theimage processing apparatus according to claim 6, wherein the CPU isfurther configured to generate viewpoint information indicative ofpositions of the first viewpoint and the second viewpoint.
 8. The imageprocessing apparatus according to claim 7, wherein the CPU is furtherconfigured to: generate a plurality of pieces of the viewpointinformation indicating a plurality of candidate pairs of positions ofthe first viewpoint and the second viewpoint; generate, for each of theplurality of candidate pairs of positions of the first viewpoint and thesecond viewpoint, texture images of the first viewpoint and the secondviewpoint; set the texture images corresponding to a specific candidatepair of positions of the plurality of candidate pairs of positions ofthe first viewpoint and the second viewpoint, as final texture images ofthe first viewpoint and the second viewpoint, wherein the set textureimages, corresponding to the specific candidate pair of positions of thefirst viewpoint and the second viewpoint, has a largest region of aspecific rear face of the polygon among the texture images correspondingto the plurality of candidate pairs of positions of the first viewpointand the second viewpoint; and generate depth images individuallycorresponding to the final texture images of the first viewpoint and thesecond viewpoint.
 9. The image processing apparatus according to claim8, wherein the CPU is further configured to determine the plurality ofcandidate pairs of positions of the first viewpoint and the secondviewpoint based on successive rotation of a straight line by a specificamount in at least one direction around the center of the polygon, thestraight line interconnects the first viewpoint and the secondviewpoint, and the straight line passes through the center of thepolygon.
 10. The image processing apparatus according to claim 8,wherein the CPU is further configured to output the plurality of piecesof the viewpoint information individually corresponding to the finaltexture images of the first viewpoint and the second viewpoint.
 11. Theimage processing apparatus according to claim 6, wherein the CPU isfurther configured to generate the polygon based on picked up images anddepth images corresponding to the picked up images, the picked up imagesare acquired by a plurality of cameras which is around an imagingobject, the imaging object corresponds to the polygon, and at least apart of the imaging object is in an imaging range of each of theplurality of cameras.
 12. The image processing apparatus according toclaim 11, wherein the CPU is further configured to: generate a thirdtexture image of an omnidirectional image based on the picked up images;and generate a third depth image of the omnidirectional image based onthe depth images corresponding to the picked up images.
 13. An imageprocessing method, comprising: in an image processing apparatus:generating each of a first texture image of a first viewpoint and asecond texture image of a second viewpoint, wherein the generation ofthe first texture image is based on projection of by projecting a firstrear face of a polygon to a first projection face with respect to thefirst viewpoint, the generation of the second texture image is based onprojection of a second rear face of the polygon to a second projectionface with respect to the second viewpoint, the first projection face isperpendicular to a first sight line direction, the second projectionface is perpendicular to a second sight line direction, the first sightline direction is from the first viewpoint to a center of the polygon,the second sight line direction is from the second viewpoint to thecenter of the polygon, the first viewpoint and the second viewpoint areopposed to each other across the center of the polygon, the firsttexture image corresponds to a first reconstructed portion of thepolygon, the second texture image corresponds to a second reconstructedportion of the polygon, and each of the first reconstructed portion ofthe polygon and the second reconstructed portion of the polygon isgreater than one-half of the polygon; and generating a first depth imageand a second depth image, wherein the first depth image corresponds tothe first texture image and the second depth image corresponds to thesecond texture image.
 14. An image processing apparatus, comprising: acentral processing unit (CPU) configured to: generate each of a firsttexture image of a first viewpoint and a second texture image of asecond viewpoint, wherein the generation of the first texture image isbased on projection of a first rear face of a polygon to a firstprojection face with respect to the first viewpoint, the generation ofthe second texture image is based on projection of a second rear face ofthe polygon to a second projection face with respect to the secondviewpoint, the first projection face is perpendicular to a first sightline direction, the second projection face is perpendicular to a secondsight line direction, the first sight line direction is from the firstviewpoint to a center of the polygon, the second sight line direction isfrom the second viewpoint to the center of the polygon, and the firstviewpoint and the second viewpoint are opposed to each other across thecenter of the polygon; generate a first depth image and a second depthimage, wherein the first depth image corresponds to the first textureimage and the second depth image corresponds to the second textureimage; determine a plurality of candidate pairs of positions of thefirst viewpoint and the second viewpoint based on successive rotation ofa straight line by a specific amount in at least one direction aroundthe center of the polygon, wherein the straight line interconnects thefirst viewpoint and the second viewpoint, and the straight line passesthrough the center of the polygon; generate a plurality of pieces ofviewpoint information indicating the plurality of candidate pairs ofpositions of the first viewpoint and the second viewpoint; generate, foreach of the plurality of candidate pairs of positions of the firstviewpoint and the second viewpoint, texture images of the firstviewpoint and the second viewpoint; set the texture images correspondingto a specific candidate pair of positions of the plurality of candidatepairs of positions of the first viewpoint and the second viewpoint, asfinal texture images of the first viewpoint and the second viewpoint,wherein the set texture images, corresponding to the specific candidatepair of positions of the first viewpoint and the second viewpoint, has alargest region of a specific rear face of the polygon among the textureimages corresponding to the plurality of candidate pairs of positions ofthe first viewpoint and the second viewpoint; and generate depth imagesindividually corresponding to the final texture images of the firstviewpoint and the second viewpoint.